DOCOHEHT RESOH? 



BD 163 9.70 ^ 

TITLE • 



IB 006 641 



INSTITOTION 
SPONS AGENCY 
r HEPORT. NO' 
^' .PUB PATE 
" ~ Cd¥!BRA C!er " 
NOTE - 

^.SDRS i^EICE - 
DESCRIPTORS- 



V 



-Kaskoiritz^ David; ^'Suppes, Patrick y • ? .. 

Development and "Impi'e^mentation of :^Comp^ter Model 
for Student Management: 'Phases I airS II. . Interim . 
':ReportJ "April 1977-December 1977. 
Stanford Oniv,, Calif • V, 

Air Force Human Resources Lab., Brooks AFB, Texas^ 
AFHRL-TR-78-7 ' 

Mar 78 . ^ ' ' . ^ ' 

F33615^77-C-D04T — : - r " 

Sip. • ■ 

MF-$0^ 83 HCr$3^0 PIus>Postage. 

.^Computer Manaj^M. Instruction; Continuous Progress^_^ 
. Plan; -♦CarriculS-m Development; Educational Research; 
^ y Goodness' of Fit;. ^Mathematical Models; Military 

-Trai,aing; Predietive Validity; Predictor Variables; 
St ^Jti^tical, Analysis; *Technical Education 

ABSTRACT ' ^ .\ ' - . ' - ' . 

Re suits, are "described: for the. first tvo phases of a 

. stu.dy to develop and e^valaate models of student progress in a 
technical training/ coarse, beiiig -.of f ered in a^ , computer managed 
instructional environment' at -Lbwry Air 'Force* Base, . Colorado. Se^ecal 

.categories of models ..Were developed in Phase I and defined by model 
type and by the typ^ of* variables used as predictors. In Phase II. the 
models were evaluated using data collected on 368 students in an 
inventory management course. The evaluation consisted of parauaeter 
estimation and derivation of goodness of fit sitatisti^cs. The results 

^ indicated that by using the performance information on the initial 
blocks of the course^ >m6re pr^cis^^ pre,dictions of course completion 
time can be made than .when only; gr^assessoent ' data' are used., 
(Author/CMV)- J . ^ 



* Reprodoictions supplied by EDRS are thf bes^t that can- be made * 

■ \ ' from the original documents -* , ^ ^. * 



A 



ERIC 



H 
U 
M 
A 
N 



R 
E 
S 
0 

U 
R 
C 
E 

S 



U 5 OEPARTMEMTOf tic«*.in, 
EDUCATION!. WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS BEEN REPRO- 
DUCE DEx'i.OUY AS DECEIVED PROM 
TnE PERSON OR ORGANIZATION ORIGIN- 
IlfNG IT POINTS OF yiEV* OR OPtNlONS 
STATED DO NOT NECESSARILY REPRE: 
SENT OP PICIAU NATIONAL .NSt.TUTE OF 

EDUCATION POSITION OR POLICY 



PEVELOPMENT AND IMPLEMENTATION 
OF A COMPUTER MODEL FOR 
STUDENT MANAGEMENT: 

PHASES I AND II 



By 

David Kaskowitz 
Patrick Suppes 
Stanford UniversitY 
Stanford, California 94305 



TECHNICAL TRAINING DIVISIOM^ 
^Lowry Air Force, Base, Coloraao 80230 



March" 1978 

Interim Report for 8eriod April 1977 — December 1977 



Approved for public release; distribution unlimited. 



LABORATORY 



NOTICE 



When U.S. Government 'drawings, specifications, or other data are "used 
for any purpose other than a definitely • related Government 
.procurement operatiort, the , Government ' thereby ' incurs no 
responsibility nor. any . obligation whatsoever, and- the feet that the 
Government may. have formulated, furnished, or in any way supplied 
the said drawings; specifications, or other data-is not to bp regarded by 
- imjiicatiori' or otherwise, as in. any^^ 

other person or corporation, or conveying arijf rights or permission to . 
mffliufacture, use, or sell any patented invention that may in any wa^'. 
be related thereto. , * ^ - 

' &^ > - . 

This interim repo«f was submitted by Stanford University, Stanford, 
California 94305, under contract F33615-77-C-OM1, project 1121, 
with 'Technical Training Division, Air RWce Hiiiuan Resource^ 
Laboratory (AFSC), Lowty Air Force Base, Colorado 80230. pr. Roger 
J- PenneD, Instmctional Technology Branch, was the contract monitor, 
■f^ * ■ ^ 

This report has been reviewed and cleared for open publication and/t)r 
pubhc rele^ by the appropriate Office of Information f (01) in 
accordance with AFR' 19&-17 and DoDD 5230.9. There is no objection 
to unlimited distribution oif this report to the public at large, or by 
DDC to the National Technical Information Service (NTIS). 

This technical report iias been reviewed and is approved for publication. 

^ MARTY R. R0CK5V AY, Technical Director ^ i> 

^ Technical Training Division 

DAN D. FULGHAM; Colonel, USAF 

Commander , J 



unciassuiea . 



SECURITY CLASSIFICATION OF THlS'PAG^ (WffO Omtm tfitcred) 



REPORT DdCUWENTAT-IObI PAQE 


READ INSTRUCTIONS . 
' • * BEFORE COftl^LETING FORM 


1. REPORT NUMBER • , 2. GO>Vt ACCESSION" NO. 

AFHRL-TR-7i-7' ' . 


3. RECIPIENT'S CATALOG NUMBER . * 


4. TITLE (ana Subtlttp) - . . v 

DEVELOPMENT AND IMPLEMENTATION OF A 

COMPUTER MODEL FOR STUDENT MANAGEMENT: - • 

PHASES I AND U 

• ' . ■ ~V -V.' ' . ■ 


5. TYPE OF REPORT ft ^ERIOO COVERED 

. .Interim^ 
April" 1977 -December 1977 


6v PERF.ORKING ORG. REPORT NUMBER 


7. AUTHORC*; ' ' ' * 

. David TCaskowitz ' . ' . ♦ • - 
Patrick Suppcs ♦ - ^ v v ^ 


B. CONTRACT OR GRAI^T NUMBERfa; 

'.P33615-77-C-0041 , . . 

■ ^ T 


9. PERFORMING ORGANIZATION NAMC AND ADDRGSS . ' C 

Stanfopd University . • • , 
Stanford, Califomia'94305 ■ ^ ■ ^ 

' * • ■ ** ' , ■ " 


-JO. PROGRAM ELEMENT>?R0JECT; TASK 
AREA ft.WORiC UNIT I^UMBERS" 

^62205F . ^ > ' r 
11210220 , ■ . , 


11 - CONTROLLING OFFICE NAME AND ADDRESS . 

HQ Aip-Force Human Resources Laboratory (AFSC) 

Brooks Air Force Base, Texa3»7 823 5 * ^ 


,12. REPORT DATE 

. March 1978 A . 

: i z : _ 


13 NUlHBER OF PAGES 

62- ^ : / 


i4. MONITORING^AGENCY NAME ft ADDRESSf'/f £i///or«nf from ControlUng OWce) 

Technical Training Division 
Air Force Human Resources Laboratoiy 
^ Lowry ''Air Force Base, Colorado 80230 , ' ' 


IS. SECURITY CLASS. C«( ^'•^'•P^*'^^ 

% Undas^^ied ^ ' ^ 


ISa. declassification/ DOWNGRADING^ 
SCHEDULE- 


16. DI ST RI BO TI ON STATEMENT fof RoporO . , ' 

Approved for public release; distribution unlimited. ' ' . - 


17. DISTRIBUTION STATEMENT (of the nbstract entered in Block 20, if dlfforent Itom Report) 

y^vr.-^' " . ■ 


ija^.SlipPLEM^NTARY NOTES . " 

^ Y * ■ ' " • ' ' ■ * - ^ 

' ^r ' . ^ , ■■ • . . , . ' - - . " 

: ' " ' * ' . ig '■ : 


KEY-WORDS (Continue on te^rao aido if nocoaaary and identify by block nxjmber) 

■ ■ ' . . ■ . . ■ ^ • ^ • ■ 

cRjniputer managed instruction j _ ' . 
cotW^ completion times \ . ' . , ' 
rnathernatical models - . * \ , ' ' 
sradent prbgcess management - 


20. ABSTRACT (Continue on rovorao aide if nocaaaary and identify by block number) J 

This report describes the^restilts of .the first two phases of a study tc^ develop and evaluate models of student 
progress on a technical training, course being offered in a computer managed instructional environment at Lowry Air 
Force Base, Colorado. Several categories Qf models were developed in Phase L The cat^ories were defined by model 
type and by the type of variables used as predictors. 

. In Phase II, the models \vere evaluated using data collected on" 368 students in an inventory man^ement 
course. The evaluation consisted of parameter estimation and derivation of goodness of fit statistics. The results 
indicated that Jby using the perfcfrmance information on the initial blocks of the course, more precise predictions of 
course comfietion time can be made than when .only pieassessment data are used. . . 



•QERiC"? 1473 eoiTioN OF t NOV 6S IS oBsouETE _ ' ' . ' " Unclassjfied 



. ' ' . STtCURITY CLASSIFICATION OF THIS PAGE (T»J>on Ca/a £nf«wod3 

' -9 ' ' . : 



AFHRL-TR-78-7 



AFHRL-TR-78-7 



PREFACE 



\ 



The authors wish to acknowledge the assistanc^ received from the staff at SRI International who 
worked on this project under, a subcontract to Stanford University. Eh". John Draper served as SRPs 
project supenrisor. Mr.^odl^erirhbmas served as SRI^ project leader in the early portion of the project. 
Msr-Fran-Adams irode-a mijor/contribution-to the fiteratu^^ 



• . • . TABLE OF CONTENTS 

. • A ' 

UST OF TABLES .' .4 ' 

I INTRODUCTION . . . *. 7 

Background 7 

Overview of the AIS :w 8 

-~ II DESCRIPTION. OF THE MODELS AND THE METHODOLOGY : r ; 7 7 ... .7 . : : . "T . T". T 9 

Description of the Models of Student Progress ^ ! : 9 . 

The Data Base 'r /. ' 11' 

Procedure for Creating the Achievement Index 16 

; Parameter Estimation , . . • 17 

Evaluation Measutfl^ . . . . .* 17 

III ^SULTS . . . . . . . . ' : 19 

Results for'BMMl '\ . 1.9 

Resuhs for BMM2 ^ . . . ? 19 

R^suluforPMM *. , . 19 

Results for BTMl , '23 

Results for BTM2 , . . . : 25 • 

Results for PTM . . * . . 30 

Results for MMM 30 

Comparison of the Baseline Models , ^. . . ; . . 32 

^ Comparison of the Performance Models . , ^.^ . . 32 

IV* CONCLUSIONS AND RECOMMENDATIONS , .\ • 34 

Conclusions Based on the Results of the Evaluation \ . 34 

Other Considerations ; ... .L ..... : '. 34 

Recommendation for Phase III . . .\ . , . ^ : . I 37 ^ 

REFERENCES /. ^ .\ . / . . 39 

.APPENDICES \ ^: * .'.43 

A UTERATURE REVIEW " , 43 

* ' B SUMMARY STATISTICS . . : v. ^ 56 




. UST OF TABLES. : 

r Number of Lessons per Block of Instruction in / ^ 
the Inventory Management Course ... 

2 "Statistics on Edit of the Hecent Data FUc . . . ^ 

3 Variables Included in the Analysis File . • . . . 

4' Differences Between Cumulative Block Elapsed Times 
and Corresponding Cumulative Lesson Times to 

Criterion . . • 

5 Comparison of the Achievement Index and Nonnalized. 
Block Times \ . ; ^ . 

6 Sumniary..Statistics for the Regression of the 
Cumulative Block Elapsed Times on the Baseline 

Variables ^ - - . 

N . ' * / 

7 Summary Statistics for the Regression of the 
Cumulative Lesson Elapsed Times on the Baseline ' - 
Vaii^ble^ . . ' - • 

8 Statistics from the Regression of Cumulative ~ . - 
Lesson Time on Baseline Variables for Selc/cted 
Lesson 

9 Statistics from the Regression of Course Time on ^ ^ 
Initial Blocjc or Lesson Times for the Performance' ' ^ 

'Milestone Model » . : 

TO Statistics from the Regression of Remaining Course • ? , >^ 
Time on Initial Lesson Times for the Performance 
: Milestone Model for Selected Lesson T ...... - . .... . - . : I " 

1 1 Goodness of Fit Statistics for the Linear Trajec- . . 

tory Model (BTMl) at the Block and Lesson Level . . 

P2 Trfean and Standard Deviation of th^Estimated ' <; 

. Parameters for the Baseline Trajectory Model 1 . . 

! ..'v . " ' ^ ■ ' • ^ " ' i . ' 

13 Stathtics from the Regression of the Linear Trajec- ^' / ' ' ' " * ^ ^ , • ' 
tory Parameters on the Baseline Variables * . . . . 

14 Statistics on the Estimate oTlhe^BCoefl^cient as * ^ ^ ' 

a Function of K .\< • . 

1 5 . Goodness of Fit Statistics for the Nonlineaf Trajec- " [r 

tory Model (BTM2) at the Block Level .v. . . .\ . 



16 Nkan and Standard I>viation of the Estimated . . - 
Parameters for the Pterformance Trajectory Model . / .29 

17 Statistics on the' Regression for the Mixed " * » 

Milestone Model * 

18 Goodness of Fit Statistics for the Three. Baseline ' .iir, ' 
Models: Block Level Analysis 1 , . - r*. . v — -31 

19 Goodness of^Fit Statistics for the Three Baseline ; ^ 
Models Lesson Level Arialysis; at the En3 of - 
Each Block : . . J . . . .\ . . . ^ v .1 . 33 

20 Goodness of Fit Statistics for the Two Performance 

Models: Block Level and Lesson Lex<?l ^ 34 

•n . *• ' 

■s 

/■ — 

1 ' - » 



/ 



< 



ERIC\ 



AFHRirTR^78.7 



DEVELOPMENT AND IMPLEMENTATION OF A COMPUTER 
MODEL FOR STUDENT MANAGEMENT: Phases I and U 

I INTRODUCTION^ 



Background : ~, — -- 

One of the major advantages in computer-assisted instmction (CAI) and computcr-nianaged instruction 
(CMI) is that they can be individualized so that students may proceed through a curriculum at their own 
pace. In contrast to conventional instructional organization, \^ere students proceed in a lockstep fashion, . 
a CAI/CMI^ environment enables students to proceed-at apace consistent with their own abilities and 
motivations. Thus, slower students can have time to master material arid will not get lost or retard the 
progress ot other students. Similarly, faster students will tend to get less bored when they can proceed at 
their ^wn pace. By allowing for self-pacing. In instructional program gives students more autonomy and 
can be more efficient than conventional ptograms that may necessarily be geared to the pace of the slower ' 
. students in the class. 

Although self-paced instruction has many advantages, it docs introduce some problems needing 
solution. Notably, if students are not given information regarding standards of performance, they may be 
at a loss as to how to pace tl>emselves through a course. A totally self-paced system can create administrative 
problems as well if certain actions necessary at the completion of a course require preparation days or weeks 
in advance and there is uncertainty aS to when a student will complete the course. Furthermore, for CMI or 
CAI to be run efficiently, it is important that students are progressing at a pace consistent with their ability 
and that students who are not progressing in such a manner are identified and appropriate remedial action 
is taken. Remedial action may include provision of special assistance or application of incentives tied to 
course progress. On the other hand, a schedule of incentives for rewarding exceptional rates of progress 
through a course also requires that some criterion be established regarding student progress. 

A s^dent progress management system (SPMS) can enhance the effectiveness of self-paced instruction 
by providifo information to students, instructors, and administrators on expected and actual rates of prog- 
ress throu^u. course of instruction. The studwit is informed of what is expected, the instructor is provided 
information necessary for monitoring studeirt progress, and the administrator is provided information 
necessary for planning outprocessing activities. At the heart of such a system are the procedures by which 
expectations of student progress are determined from baseline and initial performance information on the 
student. The. individual identification of expected perfomiaifce maintains the individualization of the insti:uc- 
tional system while imponng enough structure for efficient'operation of the system in terms of optimizing 
student flow. ^ . . 

' The Technical Training Division, Air Force Human Resources Laboratory^^HRL), Lowry Air Force 
Base, Colorado, funded this study by Stanford University to develop and implement a mo^l of student 
progress as part of a student progress management system in the Advanced Instructional System (AIS) l 
implemented at Lowry. This report describes the results of the'<Iesign phases (Phases I and^II) of the study > 
and includes recommendations for tasks to be carried out during Phase III. Phases I and n consisted of a ^ 
review of pertinent literatiire, formulation of alternative models of student progress, evali^on of the. 
models using actual data collected by the AIS, and recommendations for a system to be implemented during 
Phase m/ . ' ^ 

Section II contains a description of the models of student progress that were examined and a descrip- 
tion of the methods used in the ^Valuation of models; Section III contains a description of the results;, 
finally. Section IV contains the conclusions of the study and recommendations foflPhase m. In the remain- 
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der of this chapter is presented a brief description of the AIS implemented at Lowry AFB, including the 
ci^rrcnt student progress management system. The detailed report of the literature review is contained in 
Appendix A. \ . 

r 

Oversew of thi^ AIS 

The AIS at Lowry Air Force Base now includes four courses: Inventory Management, Materiel Facil- 
itiesi Precision Measurement Equipment" ajnid Weapons Management/Each course consists of a series of 
lessons grouped into blocks of instruction.- To complete a lesson or a block of lessons, a student must 
; satisfactorily cornplete a lesson or block criterion progress check consisting of a written test, a performance 
^; test, or instruction certification. Tor, some lessons a number of a/ternative modules of instruction are avail- 
able that differ in modd. and level of presentation. 

Student assigrunents to lessons and modules are con^trolled by a computerized component of the AIS 
called the Adaptive Model. Assignments to lessons and modules are based on student characteristics and on 
material and personnel utilization in orde^^ to optimize overall system efficiency by nunimizing predicted 
^ student course time and maximizing the use of course materials and personiieL As a resiflt, the sequence of 
lesson presentation and the type of module presented for a given lesson vary across s^dents. The sequence 
' with which blocks of lessons are presented, however, is usually fixed across students. 

The current AIS system includes a student progress management component (Dallman,^977) ^at 
^ was pilot tested early in 1977 antf^^^ly implemented begimiing in July 1977. Th^predictive model i&ed in 
the student progress management corn|)onent consists of a linear regression of course completion time on 
baseline variables consisting of student demographic characteristics and'preassessment test scores. Predicted 
course completion time can be converted to targeted course completion time by a policy function which 
can truncate extremely low or high predicted course completion times and can reduce or raise predicted 
course completion times by a fixed percentage. For example, at the present time predicted course comple- 
tion times are decreased by a fixed percentage to take into account the two hours per day th^t students are 
to devote to study outside of the learning center. Once a targeted course completion time is calculated, 
targeted time on a particular block or lesson is calculated by using the proportion of mean course comple- 
tion time accounted for by the mean time on the block or lesson. 

At the beginning of a course, a student is instructed on how to make, his own Course Completion Map, 
which plots his targeted completion times and allows the student to keep track of his daily progress in the 
course. Two checkpoints are indicated on the Course Completion Map: at tfee end of block 2.aj|d 8 days 
Ijefore the scheduled course completion date. The student has the responsibility to see his instmcjtor at 
the$e times to verify that he is progressing satisfactorily, and, in the case of the graduation date checkpoint, 
to set a graduation date on the calendar. 

If a student's actual time to arrive at some point in the course is greater than his targeted time by a 
specified amount (2 days), the student is to initiate a Progress Counseling Session to confer with the 
instmctor regarding problems and potential solutions. Unless a student can convince the instructor that his 
^targeted times are incorrect^ the student and instructor will enter into a "performance conttect" that 
specifies 4he date on which the student will get back on the origin^ schedule. 

In the next section, we turn to a description of the specific models of student progress thai we evalu- 



ated and the methods used in the evaluation. 
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II DESCRIPTION OF THE MODELS AND THE METHODOLOGY 

\ 

Description of the Models of Student Prdgress 

Very f^ mathematical models of student progress were found in the process of the literature review 
(see Appendix A). In addition, past studies (Larscn, Markosian, and Suppes, 1977; Wagner et al., 1973; 

Malone et<il.;\i977) indicated that Tclativdy simple-mathematical models tend to pro^ 

tions of-.student performance. Therefore, the evaluation was restricted to the trajectory and milestone 
models described below. These models were considered to be the best candidates based on the fmdi: ^ of 
the literature review. ' \ 

A trajectory model includes a hypothesized relationship between cumulative study time and cur:/ • 
tive achievement** Cumulative achievement is a construct representing a student's location in the curriculum. ^ 
It is generally measured on some index representing average student performance. If A I'epresents the value 
of the achievement index and T represents cumulative time, a trajectory model hypothesizes a functional 
relationship between T and A: r . J ^ 

T ^ f(A; a^ , a2» • * • » ^ - » 

where the Z\ are paramet^fsTBy' estimatiii^^ ^^.z parameters, on^ttempts to describe or predict a student *s 
trajectory through a course. v 

In a milestone model, cumulative tirne to any particular milestone in a course is predicted directly, 
^ without an achievement index as an intervening variable. Course milestones would consist of particular 
identifiable points of interest in the curriculum, such as the completion of particular blocks or lessons. 

Models can be characterized by the kind of data used for predictions as well as by type. Baseline 
models employ baseline variables that are available before a student starts on a course. Demographic infor- 
mation such as age and sex, academic history such as number of years of iCliooHng and highest degree 
attained, andjjcores on cognitive and affective.tests administered before a studc;::'';: entry into the course 
may be included ^s baseUne variables. Performance\models employ data regardir. . ^ student's initial perform- 
ance on a course to predict subsequent performance. Perfc^rmance variables iiiay consist of time to partic- 
ular criteria or scores on tests. Mixed models, where both baseline and performance variables are included,- 
may also be formulated. 

The specific models reported in the evaluation are described below under five headings: 

Baseline Milestone Models (BMM) 
Performance Mileslo.ie Mo.^. (PMM) " ' 
Baseline Trajectory Modeh ''BTM) ^ ' 
Pe'fformance TrajL-ctory Model (PTM^^ ' 
Mixed Milestone i/iodel (MMMJ 
In the description of the models the following notatiol wiD be used : / 
Xj^ = baseline variable k for student j. 

tij = the cumulative time in the course for student j to contplete the ith inilest one. 

Aij = the cumulative achievement index for student j after completion of the ith milestone. 

tjj =.the predicted cumula^ve time in the course for student j to complete the ith milestone. 
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Baseline Milestone Models • ^ 

BMMI — Regression of Cumulative Times lo Achieve Milestones on. Baseline Variables, Tlie model 
^ • hypothesizes a linear relationship between cumulative time to achievc^each milestone and the baseline 
, ' viiriablcs. For each -milestone, say i, we would have: \ v * 

tij = aj + S bii^Xkj . 



BMM2 — Regression of Course Completion Time on Baseline Variables. This model is identfcal to 
BMMI in using a linear model to predict course completion time. Time to complete a particular segment 
of the course, however, is predicted^by using tljpttkio of mean lime lo compfeie the segment to mean 
course completion time. If there are n milestones in the course, then course completion time is expressed as 

/For any particular milestone i let p; = Mi/Mn* where . ^ 

Mi = the mean cumulative time to achieve milestone i 
the mean course completion time. 
Then this model predicts that for milestone- i < ^ 

^ ij ~ P i ^ n j • 

Per formance Milestone Model ^ . ' 

PMM — Regression of Course Completion Time on Performance Variables. The performance milestone 
model uses the cumulative times on the first i milestones to predict the curnulative course time: 



tnj = + S bmitn3j 
m= 1 



- Baseline Trajectory Models - * ^ ^ 

BTMl — Linear Trajector)\Model. The linear trajectory mod'el hypothesizes a linear relationship 
between cumulative.tim6 and an achievement index: 

tjj a^ + '^i^ij - - N 
In turn, it is assumed that aj and bj have a linear relationship to the baseline variables: , 

= c + sdfcXkj \ : ' ^ 



k 



BTM2 — ^5onlinea^ Trajectory Model. The nonlinear trajectory model hypothesizes a linear relation- 
ship between some power. of cumulative time and an achievement index; 

(?ij)'^J =aj + bjAij .• . • • * 

o ■ . > -10- 12 ' 
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' Again,r^he parameters Kj, aj, and b/are assumed t6;have a^linear relationshii5'to the preass^sment sc6res:. . 

^ Perfofmarice Trajectory Mod'd . \ * ^ C ' i ' ' * 

. ?TM -^line^ Trajectory N^ddeL This model correspond^c^.BTMl in hypbttjesizing a linear relatipnr 
'ship between cu?nulative time-^d cumulative achievement. hd|^ver, in^this case, the-parajnete^^j and* . 
^'l a;e^estmiated*dire ^sing.the first I'obseTrved cumulative times, piejcourse completion time tn> for ^ 
Studeftt Us predicts to be: - . - /' ^ , * 't.: ^ 



4: ^ 



^ < 




using the times on the -first i milestones to estimate aj^ and bj^ 

\ . ■ ■ ■ ■ - . ■ ■ 

\ • • • . ■ ■ 

J^ixed Milestone Model ; - ■• ' * " . . , 

* the mixed milestorieYncydel (MMM) uses th^^aseline variables,>s well as'the cumulative times oh the 
first i nfiilestones to predict course completion time: ^ 



^ m — I K 



Other Models 



Several other models were considered, but complete evaluations were not conducted because in te r- 
mediate. results indicated that they were not viable altemaXives. As^is pointed out in the hierature review 
rin Appendix A, the results of Malone et al. (1977) in evaluating the predictive fit of various trajectory 
models indicated that the relatively simple models had the best predictive fit. Therefore, the models 
Included in the evaluation were the more simple models that appeared to have the greatest chance of pro- 
viding a good predictive fit. " . ' 



The Data Base 



Data and co^esponding documentation were obtained for the Inveijtory Management (IM) course as 
it had been implemVnted in late 1976 and early 1977. The data consisted of 1 5,259 records from the 
Recent Data File (RDF), where each record contained information for a particular student on a particular 
block of instruction. It should be noted that the data used in the evaluation of alternative models were 
collected in the absence of a student progress management system. At the time of data collection, the SPMS 
was in a pilot phaserand not enough students had completed the course under the SPMS to permit lise of 



their data in the ^hiation. 



The IM course is organized into six blocks of instruction. -Each bloi:k consists of a series of lessons, 
foUowed'by a period during which the student reviews the material oh the block and takes the block test. 
With the exception;of the first block, "the lesson prior to block review consists of a **chief of supply lab." 
If the chief of su^V labs and block reviews are counted as lessons, there are 61 lessons in the course, 
organized as shown in Table 1 . / . > 

. Each re^iordidfrom the RDF contained data for a particular block of instruction or contained summary 
data for thfe eiitire course (block zero records). After preliminary tabulations of the data, a file was created 
that consisted oiF data on 760 students wHo had entered the IM course during 1977 and had reliable block 
elapsed time' data for all six blocks of instruction. Reliable block times are defined as times resulting from 
actual real-tiine system tracking:' Ttfble 2 shows the number of records retained at each step of the process - 
. fey wWch this file was;created. , - ' 
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Table 1 



NUMBER OF LESSONS PER BLOCK OF INSTRUCTION, 
• IN THE INVENTORY MAIJaGEMENT COURSE 



. . Block . Number of Lessons * 
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' <!. . ^ i ., "• -. ■ V 9 

■ •■ ■ ■ . X - 

f ' . ^ 2 ' ' ■ ■ " "10. 

*r r •• • '. ' ' 3 ' hf' 12 

•■■ ' ' • 4 •* • : ■■•9 ■ 

• ' V. ' ' i ■ - .1-2 

6 ■ . 9' 

• * • . . . 

- ■ . Total ■ 61 
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" * . J'able 2 / * 

STATISTICS dN EDIT OF THE RECENT DATA FILE , T 

Number of records on original file 15,259* 

* 

Number of students represented" , ^ 2,539 

Number of students rejjres^ntedfwho entered the course in 1977 99^ 

Number of students who entered the course in 1977 who had ' / 

records for all seven blocks 766 

Number of above who had all preassessment data available . . 760 

Number out of 760 students with all block elapsed times 

(BELTs) reliable . . 550 



As was recommended to us, only students whp had entered the IM course during 1977 were 
included in the evaluation. Also, students had to have finished the course, as indicated by the presence of 
all seven blocks of information on the RDF, in order to be included. Since cumulative time to finish each 
block was/a critical variable of interest, it was decided toMclude only those cases where all the block times 
were reliable. This resulted in deleting about 30% of theses from the evaluation. The distribution of 
included students by course version is shown in Table 3. 

' The tabulatioa indicates that the great majority of students were in course version 1 , This was the 
' course version where the Adaptive Model determined lesson sequences. On particular lessons, it .determined 
the module to'te assigned from a set of alternatives. Also, the student process management system was 
not implementaliaxourse versibn 1 at the time the data were collected. The subsequent tabulations will 
include only the ^^for the 368 students in course\version 1 with all block elapsed times reliable. 
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DISTRIBUTION QF STUDENTS BY COURSE VERSION 



^ Table 3 ^ 



^ Percent 

Cburset Version ' BEL^frReliable ' ^ of Total total Number 




. • ^ ^ 36« ' . -72 ^'^^ ^ 

2' , : _ : • 10 59* 

3 ^ ■ 33 '70 

4 . ' ^ 43 -'77 
i 5 52 76 . ^ 

6 . " • 44 - , 69 

. ^ ■ : ■ ■ ■ ^ .'■ . > ^ 

The file included the .variables listed in Table^4. All^times on the file were Expressed in minutes. In the 
creation of this file the following -conventions were followed: 

• If Meaisured Time to'Lesson Criterion (LTMC) was greater than 600 minutes or if the flkgs on the 

" RDF (LDPF and MLTR) indicated that LTMC was unreliable, theti an imputed time was qalculated 
by multiplying the mean LTMC for the lesson by the ratio of the'^m of tl^e student *s available ^ 
LTMCs to the sum of the corresponding mean LTMCs/ ; ^ . " . • 

/ ' \ ' 

Of the 760 students on the file, 81% had at least one LTMC that had to be imputed. Jhe 

mean number of imputations was 4.6. Of all the lesson times, 8% had to be imputed- OF 
the 368 students in' course version 1 with all block elapsed times reliable, 68% Had at" 
least one LTMC that had to be imputed. The mean number of imputations was 3-0, rep- 
resenting 5% of the LTMCs for this group.' 

• The sequence of lesson presentation was determined using the date and time of day 
(LCDT and LCTM) that the lesson criterion was met as indicated on the RDF. If LCDT 
or LCTM was missing or if the sequence of lessons for a block was inadmissible as 
indicated by the course hierarchy charts, the most common lesson sequence for the 

^ - block was assigned to the student. 

Of the 760 students pn the file, 83% had at least one lesson sequence imputed on the 
first four blocks. (The last two blocks had a fixed sequence of lessons.) In all, 41% qf 
the lesson sequences on the first four blocks were imputed- Of the 368 students in . 
course version 1 with all block elapsed times reliable, 77% had at least one lesson 
sequence imputed on the. first four blocks and a total of 36% of the lesson sequences^ 
on the first four blocks were imputed. Although these percentages of imputed 
sequences are high, the effect on the analysis is probably negligible since sequencing 
of lessons probably has a very small effect -on tirne to criterion^ ^ 

• If a block elapsed time (BELT) was equal to zero, or if the flag on the RDF (BLTR) 

• indicated that BELT was unreliable, then BELT was considered jnissing and assigned 

a value of -1. ' ' • 

. Preliminary tabulations indicated that there w?re relatively large differences between the actual block 
elapsed time variables, definectas block elapsed timerninus absence time, and the correspj^nding sums of 
lesson time to criterion- Table 5 summarizes the differences found. : 
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Table 4 



VARIABLES INCLUDED IN THE ANALYSIS FILE 



.Variable title * 
Student LI).\ 
. Couiro Ve^on* 
Course Entry Date* 

4 

Module 'Number 
Sex*(b) 

Highest School Year 
•Completed* (b)* 

^ Student's Age at 
' Course Entry* (b) 

Reading Vocabulary 
.General Scale* (b) 

Reading Vocabulary 
Scientific Scale* (b) 

Reading Vocabulary 
total Scale* (b) 

Pre-Course State 
Curiosity* (b) 

Pre-Cour^ State 
Anxiety* (b) 

Trait Curiosity* (b) 

Trait Arpciety* (b) 

Internal-External 
Scale* (b) 

Test Anxiety* (b) 

Preference for Audio 
Mode* (b) 



Variable Label 
SI , , 
.CRSVSN 
' CRSEDT*' 



MNOIi 
SEX 

HIYEAR 
ENTAGE 
RVOCGN 



Description and Comments 



Unique LD. as^gned to_each student. • 

• Course version student enrolled in. 

. - • . '. 

' "^Date student entered course, (i.e., took 
" preassessment). , * 

Module number for 17 lesSons where alternative 
modules are available (i = I, . . , , 17). 

Code representing stuSent's sex. 

Highest school year completed. 

Stucjent's^e in years, at course entry (roundeU 
to nearest year). 



RVOCSC 
RVOCTL 
STCUR 
STANX 
• TRCUR 
TRANX 
lESCL ^ 
TSTANX 
PREFA 



Student's score on- the readingrvocabulary, general 
, scale (preassessment). * ' 

Student's score on;the reading vocaSuI^y^ 
scientific scale (preassessment). 

Stutiem^ score on the reading vocabulary total 
scale (preassessment). ' / ^ 

Student's score on the pre-course state curiosity 
scale (preassessment). ' 

Student's score. on the pre-course state anxiety 
scale (preassessment). 

Student's score on the trait curiosity scale . ' ^ 
(preassessment). . . ^ 

Student's score on the trait anxiety scale 
(preassessment). ■ . 

.Student's score on the internal-external stale 
(preassessment). 

Student *s score on the test anxiety scale ^ 
(preassessment). 

Student's score on the audio preference ^cale of ' 
the General Media Preference Test (preasse^sment). 



* Documentation on this variable was taken from '*DEP Variables List" provided by AFHRL. 
Note: (b) indicates the variable is included in the set of baseline variables. . . 
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Table 4 (Conti^iued) 



Variable Label 
PREFV 

'- PREFP 

EXPSP 

tXPCI 

READSl 
READS2 
READST 
LOGREA 
CONFIG ' 
MEMNB 
MEMNT '■■ 
' BELTi 
CABELTi 

Tli 

CBAli 

LTMCi 

CLETi 

GAIi 



Description and Corah^ents 



Vaiiable Title 

.Preference for Visual 
-Mode*'(b) 

Preference for PrinteS 
" Mode*(b) ' / 

- ' Experience with Self 
Pacing* (b) 

/ 

Experience with 
Conventionai 
Instruction* (b). 

IM/MF Reading ^ 
Subscale I* (b) * 

IM/MF Reading 
. Subscale 2* (b) • 

IM/MF Reading Total ^ 
Scale* (b) 

IM/MF Lo|?cal 
Reasoning Scale* (b) 

• Concealed Figures 
Scale* (b) . 

Memory For Numbers - 
Backward Scale*^(b) 

Memory ftr Numbers; « 
Total Scale* (b) 

Block^ Elapsed Time 

Cumulative Actual 
Block Elapsed Time 

Measured Time Absent 

Cumulative Block 
Achievement Index ; 

Nfeasured Time to 
Lessons Criterion 

Cimiulative Lesson 
• Elapsed Time 

Cnmufative Achievement 

Index . • 
^ 

^Documeniatioh on this variable was taken, from "DEP Variiables'Iist'* provided by AFHRL. 
Note: ,j(h} indicates the variable lis- in'cluded in the set of baseline variables. 



^ Student's score on the visual preference scale of the 
General.Me^dia Preference Test (preassessment). . 

Studerff s score on the printed preference scale of ' 
the General Me^a Reference Test (preassessment). 

Student's score on the experience with self pacictg 
scale of the General M^dia Preference Test 
(preassessmenf). 

Student's score on the experience with conven- 
tional Instruction scale of the General Media 
Preference Test (preassessin^t): 

Student's score on the IM/MF reading skills test, 
'- subscale 1 (preassessment). 

Student's score on the IM/MF reading skills test, 
sub^ale 2 (preassessment). 

Student's score on the IM/MF reading skills test, 
total scale (preassessment).-: 

^ Student's score on the IM/MF logical reasoning , 
Sjcjale (preassessment). 

^tudent^ score on the concealed figures scale 
''(preasses^iQeirt). ■ 

Student's score-on tfie IM/MF memory for numbers 
test, backward, scale (preassessment). . . 

. . : Student's score on the IM/MF memory for numbers 
test, total scale (preassessment). 

Regular elapsed classroom time while student was ^ 
in block (i =^1 , . . . , 6). V ^ ■ ^ 

Regular elapsed classroom time up to the com- 
pletion of the ith block excluding absence time 
(i=l,...,6). \ ^ 

Hme absent during the ith blocl^ (i= 1 , . . . , 6). 

• Value of the achievement index at the end of the 
. ith block(i= 1, , . .,6). ^ . * 

Measured tiihe spent by the'^dent on lessoai 
imtil he first passed it (i = 1, . , .* , 61).* 

Measured time spent by the student until he first 
passed the ith lesson presented to him (i = 1, 

-..,61). : . . ■ . 

Value of the achievement index at the end of the 
. ith presented lesson (i = 1, . . . , 61). 
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. « Tabled • , - • - 

. - * DIFFERENCES BETWEEN CUMULATIVE BLOCK ' 

• ^ ELAPSEO TIMES,AND CORRESPONDI>JG. ' 

. CUMULATIVE LESSON TIMES TO CRITERION . 
** - V ' • (Minutes) 



Block- • ^ - Mean Difference S.D. 



r ; 187 ' ' 266 

2 ■ ■ 53&. . . . .475 

3 1185- • ■ '839 

4 • 1538 ' ' 978 

5 1968 , 1151 

6 2218 ' f 1210 



Discussion with AFHRL personnel indicated that differences are due primarily to: <©. ' 

• Inclusion of administrative lost time due to shift open and close at the block level and not 
at the lesson level. ^ _L 

• Intermittent omission of block remediation time after a failure on the initial block test. . ' 

After some further tabulations <)f such variables are Measured Materials Remediation Time (T3), iX 
was decided^ o cany out the analysis- separately on \he block elapsed-times and on the lesson time to 

criterion to see whetlier prediction using one of thes^sets was better than prediction using the other ^et. 

. 1 ' - 

Summary statistics for the variables included in the analysis are tabulated in Appendix B. These ^ 
include means and staDdard-deviations as well as selected correlations among variables. 

The ^4 variables indicated by a "b" in the "Variable Title" column in Table 4 comprised the set 
referred to as baseline (or preassessment) variables in the description of the models. A few other preassess- 
ment variables that appeared on the RDF were excluded from the evaluation because of lack of variation.. 

Procedure for Creating the Achievement Index 

An index of achieveme/it was needed for the trajectory models. After considering alternative 
approaches to defining the achievement'index, it was decided to use mean lesson time scaled by mean 
course completion time. That is, each lesson was assigned an achievement, value by dividing the mean 
measured time to lesson criterio'h by the sum of the mean me^urcd times to lesson criterion. The ratio was 
then multiplied by 100 so that the cumulative achievement index would indicate the percentage of the 
course completed at any particular time. In this way, the achievement index is linearly related to the mean 
cumulative time to complete each block, calculated by summing the appropriate lesson times." = 

Tabled shows the value of the achievement index at the end of each block of instruction and the 
cumulative block times normalized "so that total course time equals .100. Differences betjveen the achieve- 
ment index and the normalized block times are extremely small, indicating that the values of the index at 
the end of blocks would have been virtually the same if the block elapsed times had been used. 
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^ Table 6 

COMPARISON OF THE ACHIEVEMENtTNDEX AND 
' .NORMAUZED BLOCK TIMES - > 



Cumulative 
Acl^veraent 
Index > 

1-3.9 ' 
'32.2 

55.4 ' 

70.3 . 

86.9 
.100.0 



*^Block Times 
(Normalized) 

13.0 ' 
34^0 
55.5 
■'70.2 '•' 
8'^ - •- 
100.0' 



Observed Percentage- of 
"■Cumulative Lessen Time at 
the End of Each Block 
'Mean . 

14.4. 

33.1- 

55.8 

70.2 

86.8 

. 100.0 




the table also gives the mean and standard /deviation of the percentage of cumulative lesson time at 
the end of each block for the 368 students in course version 1 . The mean cumulative percentage.'of time 
spent to finish each block corresponds closely to the cumulative achievement index. 

Parai^eter Estimation * ' 

The SPSS software as operationalized on the IMSSS PDP-10 was used to generate parameter estimates. 
For the milestone models, the cumulative elapsed times served as dependent variables in a stepwise 
regression on the specified independent variables. For the baseline trajectory models^ a two-stage approach 
was used. In the first stage, parameters were estimated for each student/The estimated parameters were 
tl^n entered as -dependent variables in a stepwise regression with the ba^line variables as independent 
variables. In the performance trajectory models, the parameters were estimated for each student separately 
using the initial cumulative elapsed times, ' ^ 

The regressions that included the baseline variables were conducted in two runs. In'th^ first run, the 
stepwise regression included all 24 baseline varisEbles. The results of this run were examined and a second 
run was made that included only the most salient variable, that is, only those^6iat increased the square of ^ 
the multiple correlation-coefficient by at least .003 . This criterion was selected to reduce the number of 
variables in the final equation;it tended to reduce the number of baseline'variables from 24 to less than 
10. the criterion is rather liberal in including variables that contribute relatively little to the regression. , " 



Evaluation Measures 



Since deviation of elapsed time from targeted time is critical to the student progress management 
system, the-evaluation measures. we^ selected as functions of the distribution of residuals, defined as 
^ observed elapsed time minus predic^d elapsed time. The statistics generated for each model included: 



The mean residual 
The median residual 
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The standard deviation of the residuals . . // 

• The me^ absolute residual ** . . IP 

- • The root mean square residual. ' / ^ ' - " 

For the biascline niodels^ these statistics were generated for cumulative elapsed time to the^end of each 
block. For the performance models, they were generated for cumulative elapsed tim^ to course \ 
completion. / * - . \ ' _ ' / j i ^ 
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m .RESULTS 



^ The resuhs of the parameterjstiination are described below for each model. Comparisons of the . . 

."^ gdodness^of fit ^d predictive ac^acy of the models are thSji made. • , 

I. Results forsBNG^ ' . *- " 

. " V Table'7 qeqtains the^surrariary statistics for the regression of the' cumulative block elapsed times on the . . 
baseline variabSs>The value of the multiple corielation_^cpefficient, R2 , has-very little change between the 
"if^Sf^ith all variables entered and tlie run with only the"most salient variables etttered (see Section II for a 
descristion of the ojethodology). The v^ues stay relatively stable across blocks, with the lowest value 
of .24 fprthe first brock. The R^^^alues indicate that the preassessment scores are accounting for between 
* 24% and 30% of the valance of the cumulative block elapsed time. This corresponds to a multiple correla- 
tion coefficient of between .49 and .55. The standard errors of estimation also indicate, as might be expec- 
/ ted, that the error'in estimation increases with block number. We expect this effect because the magnitude - ^ 
of the cumulative block times will be increasing with block number. ., . 

Th^e variable/ that en^r th^: stepwise regression in the initial steps and account for most of the R2 
include the total score on the IM/MF reading skills test (READST), sex; and the score bn the experience 
with conventional instruction scale of the General Media Preference Test (EXCPI). 

• • ■ * • 

The statistics shown in Table 8 on the regression of cumulative lesson time to the end of each block 
• on the baseline scores are similar to those at the,block level. The multiple R does tend to decrease somewhat 
between the firsV^and sixth block. The' three variables that entered the stepwise reWessfon first at the block 
" level also enter first at the lesson level. Table 9 shows selected regression sfctistics aVthe end of lessons where 
. il students would^have been" presented the same set of lessons. The first few lessons in Block 1 have 
extremely low R^ values, but by the fifth lesson the value of R^ is already up to .28 . 



Results for BMM2 , , ^ . 

The BMM2 model does not require additional 'regression mns. It merely uses the BMMl results for course 
cornpletion time. ^ 



Results for PMM 



Table 10 presents the statistics on the performance milestone model at both the block and the lesson 
levels. In this case, the dependent variable is the remaining time to course completion and the independent 
variables are initial cumulative block or lesson times. For comparison, the regression statistics for course 
iletion time on the baseline milestone model are also included. • 



The column labeled "R^'.' contains the square of the multiple correlation of the course completion 
time with the initial cumulative times. The columajabeled "R^ for Remaming Course Time" contains the 
square of the multiple. correlation of the remaining course time with the ihitial cumulative times. The 
standard error of estimation is the same for both the case when course completion time is the dependent 
" variable and the case when remaining course time is the dependent variable. Under the column labeled - 
*'Mean Time Remaining" are the average times remaining in the course at the end of each block. This 
column is included for reference to indicate the ma^itude of time remaining. 

' At both the block and the lesson levels, there is a substantial increase in the R^ values between the 
baseline regressions and the regression using the performance information on the first block. Of course, the 
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' ' J SUMMARY STATISTICS FOR: THE REGRESSION OF THE CUMULATIVE 

BLOCK ELAPSED TIMES ON THE BASELINE VARIABLES ' . 



/ 



^ i 

At the end of ' * • ' 

Mc^k: - ^ . . ' ; 1 ^ . 2. ^ . -r-. 3 , ^ 4 5 ^ . . 6 

r2 with all . - . ^ «^ 

"T^ariables . 
cntexcd '^-^ \ 

r2 on the . ' 

truncated . ' , ' ^ 

run , .24 ' 30 30 ^ 30 ' ' 30 ' ^8 

Standard error ■ V 

ofcstimatiiwi ^ ' 389 - ^. 743 1231 ^ 1466 1739 1888 

Variable entered 



\-V24'' — .-^^^Tn-'* ' 30., \'..yr 30 ; ,29 -'" ' .28' 



Varia 



Step 1 


READST 


(.09) 


READST 


(.^4) 


READST 


(.14) 


READST 


(.13) 


READST 


(.13) 


READST 


(.12) 


2^ 


SEX 


(.16) 


SEX 


C20) 


, SEX 


(.20) 


SEX 


(.20) 


SEX 

* 

r ■ 

EXPCI 


(.19) 


SEX 


■4:18) 


3 


MEMNB <.18J^LOGR£a 


(-24) 


EXPa 


.(.25) 


EXPa 


(.25) 


(.24) 


EXPCI 


(.23) 


^4 


EXPa 


(.20) 


EXPa 


(.26) ] 


LOGREA 


(.26) 


logrIea 


(.26) 


LOGREA 


C26) 


^OGREA, 


. (.24): 


^ 5 


LOGREA 


(.21) 


TRCUR 


(.28) 


TRCUR 


(»27) 


TRCUR 


(.27) 


. TRCUR 


C27) 


TRCUR 


(.26) 


's 


. TRCUR 


m 


HIYEAR 


(.28) 


lESCL (.28) 
$TCUR^(.29) 


' IESCL>(.28) 


lESCL 


(.28) 


lESCL' 


(.26) 


1 


HIYEAR 


C22) 


STCUR 


(.29) 


STCUR 


(.29) 


STCUR 


(-28) 


HIYEAR 


(.27)" 


8 


RVOCTL 


(-23) 


TSTANX 


(.29)^^HIYEAR 


(.29) 


HIYEAR 


(.29) 


HIYEAR 


(.29) 


STCUR 


(.27) 




STCUR 


(.23) 


lESCL 


(.30) 


■MEMNT 


(.30) 


TSTANX 


(.30) 


TSTANX 


(.29) 


TSTANX 


(.28) 


10 


STANX 


(.24) 






r 


■ \ 
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' SUMMARY STATISTICS FOR THE REGRESSION 6?.THE CUMULATIVE 
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LMFW A ^ X ^± XlJ ± X ^Aj^ ' ^ , . 

LESSON ELAPSED TIMES ON THE BASELINE VAlUABLES • 



AT the end of 
bl6ck: 

R2,withall 
- variables ^ 
entered ^ ^ " ^ 

r2 on the;?" ■ • 
truncated. -? 

Standard error 
of estimation 

Variable entered 
{R2) 



30 

6 .29 
286 



2 
.30 

582 



J26 

25 
1149 



/:24 

23 
1336 



.23 

22 
1468 



Step 1 
2 
3 
4 

5 

. 6 
7 
8 
9 
10 




JIEADST (.10) READST (.10) 
(.14) SEX (.14) 



EXPCI (.19) EXPa (.18) 



READST (.12) READST (.12) READST (.11) READST 

SEX (.18) EXPCI (.17) EXPCI (.16), EXPCI (J6) 

. EXPCI (.23) SEX (.21) SEX (.21) SEX (.21) 

RVdCGN (.25) LOGREA (.23) LOGREA (.23) TRCUR (.22) TRCUR (,20)^;J^R (.19) 

TRCUR (.27) TRCUR (.25) TRCUR (.24) LOGREA (:23) LOGREA %21) LOGREA' 

STCUR (.27)^ TSTANX (.26) TSTANX (.25) TSTANX (.24) CONFIG (.22) CONFIG (21) 

TSTANX (28) STCUR (27) *TCUR (.26) ^ STCUR (24) TSTANX , (.22) HIYEAR(21)^ 

HIYEAR (29) HIYEAR (27) CONFIG^ (26) CONFIG (25) TRANX (23) STCUR (21) 

RVOCGN (28) PREFP (26) 

PREFP (.28) PREFV (27) 



HIYEAR (23) ^STANX (.22) 
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J- Table 9 ^ j - ; 

^ STATISTICS FROM THE REQRESSION QF CUMULATIVE.LESSOltf TIME ' ' 
ON BASELINE VARIABLES FOR SELECTED LESSONSr 

■■■'</ Standard Errbr Mean 
o- . ' ■ • - ' . . ' Cumulative ' . ' 



Block 


" > Lesson 


• ,r2 ; . . 




JLcSSOn 11X116 


1 


1 


..10 


53 


. 96' " . 




■ 2.-: 




67 . 


^ 178 . 




5 "\ ■ 


- .28 


135 


471 




13* 


.28 


286 


1033 


2 


1 


.24 - 


314 . 


1133 • 




2 


* .=29 


345 


'1263 




3 


. -29 


357 •■ 


1304 




8 


.29 


533 


2149 




12* 


\ .29 


559 


2316 




13* 


\ ,28 


582 


• 2378 


, 3 ' 


12*- 


.26 ^ 


908 


3921 




13* 


.27 


931 


4024 


4 


12*; 


.25 ' 


1120 ^ 


= 4982 




13* 


.25; 


1149 


5063 


5 . 


5 


. .24 


1241 


5571 




12* 


.23 


1312 


6133 




13* 


.23 


1336 


6250 


6 


5 


' -22 


1424 


6809 • 




13* - 


.22 


1468 


7186> ' 



*A"1 2" represents the Chief of Supply Lab; a "13" represents the block review, consisting of the bibck test 
and remediation. * . i ^ 
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Table 10 



STATISTICS FROM THE ^GRESSIORpF CO'DKSE TIME ON INITIAL 
BLOCK OR LESSON TIMES FOR THE PERFORMANCE"MILESTONE MODEL 



Block Level 





• R2- 


Standard Error 

•'. - ' of: > / „ 

* Estimation ' 


•Mean Time 
Reniaining ■ ' 


R? for 
Remaining 
Course Time 


Baseline 


.28 


4888 


^ 9405 






50 


1554 


8185 ; ' 


34 . 


2 


•74 


J116 

V 


6490 


.46 , 


3 


^ .89 


731 


4195- 


.42* 


4 


.94 


535 


2804 


39 


5 


.98 


294 o 


1187 


.21 


Lesson Level 


' 


Standard Error 
of 

Estimation 


■ 

Mean Time 
Remaining 


R2 for 
Remaining 
Course Time 


Baseline 


.22 


1468 


7186 




Block: 1 


.51 • 


1152 


6154 . 


■35., 


2 


.67 


951 


4808 


34 


3 


.84 


654 


3162 


34^ 


4^ 


• .92 


481- 


2124 


.28 


5 


.98 


258 


936 


.27 . - 



gain in for predicting course completion time using succeeding cumulative block times is necessary since 
cumulative course time is the sum of cumulative block time and remaining course time. But the R^ for 
predicting remaining, course time is also much larger than the R2 for the baseline regression. 

In any event, the standard error of estimation decreases by about 300 minutes between the baseline 
regtession and those using the performance data on the first block. The standard errors continue to 
decline at the end of each subseqiient block, indicating the rate at which increasingly accurate prediction, 
of course completion time can be made, given information on performance on succeeding blocks'. 

Table 1 1 presents the recession statistics for the performance model at the lesson level for .each 
lesson included in the analysis. By the fifth lesson, the R^ value is higher than that found In the baseline 
mil^tone models and the standard error of estimation is much less. The average cumulative time to 
acc^plish tlie fifth lesson was 471 minutes, which constituted about 7^'of the entire course. ^ 



Results for BTMI - 

The results of the parameter estimation for the linear baseline trajectory model are described in two 
parts: how well the trajectory model fit the data for individuals, and how well the baseline variables could 
be used to predict the^estimated parameters. , ^ • . 
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Table 11 "• - 

STATISTICS-FROM THE REGRESSION OF REMAINING COURSE TIME ON 
INITUL LESSON TIMES FOR THE PERFORMANCE MILESTONE 
MODEL FOR SELECTED LESSONS ' ' . 



Blocic 


Lesson 




Standard Error 
of 

Estimation 


' R^for ^ JL 
Remaining 
Course Time 




X ^ 


no 




(17 








1*K52 


•lo 








1 OCR 






r U 




1 1 D2 




r • 

2 


1 


.j2 > 








•2 




lU9o 






3 


.56 


1093 






o 

8 


.65 


9o5 






12* 


.57 


959 






13* . 


.67 


951 ' 


34^ 


■ 3 


12* 


.83 


686 






■ 13* 


.85 


654 


34:- . ' ' 


4 


12* 


.91 


502^ . 


• . " ■ ,28:-;;;.- . • 




■.13* 


.92- 


481 


.28 ■« ^ 


5 


' 5 


.95 


. 369- 


32 




12* 


,97 


: 284 


30; 




13* . ' ' 


.98 


258 




6 


^. 5 


.99 


* 143 


, .19, ; 



*A''*12" correspond^ to the Chief oT Supply Lab; a "13" corresponds to the block review. 
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The first' question is addressed by estimating the parameters of the model for each student * ' 
individually and uiing the- parame^r estimates, to generate expected cumulative times. T4ie second 
■ question is addressed, by trying to predict the estimated parameter values u^g the^baseline variables. 

Table 12 presents the goodness of fit statistics for cumulative block tiffl^and^ cumulative lesson 
time at the completion of each block, usiiig the a and4) values estimated for ea?;h student individually- 
'The goodness of fit statistics used here are the same as those used in the comparison of fits: the mean, 
stanard "deviation, median, mean absolute, and rooi mean square of the distribution of residuals, where the 
residual is defined as the- observed time minus expected time in minutes. The results indicate that the 
trajectory model provides a good descriptive fit to the data at both the block and'the lesson levels. The ' ^ 
valuespf the mean and median residuals aSe relatively small across Blocks, indicating negligible bias. The 
valu'fes of the mean absolute residual and the root mean square residual are uniformly small, indicating a 
good fit to the data. 

The mean and standard deviation across students of Ihe par^eters at the block and lesson level 
are given in Table 13. The model, as we formulated it, relates cumulative elapsed time, t, to a linear, 
fiinction of achievement. A: . - 

«» ^ ' . ' 

t=a+bA " \ - ■ 

^ ■; ■ .- I , - ■ ' ■ . 

In this formulation, the parameter_b indicates the amount of time it takes to move through 1% of the 
course material. It would be expected that the^)arameter_a_would be close to zero, and this indeed is the 
cage when it is recalled that the entire course length averages 9405 minutes using block elapsed time and 
71& "'minutes ujiing lesson ehpsed time. The differences in the magiiitude of the estimates of b. between 
the^block and lesson levels of analysis may be attributed to the differences between the two levels in the 
estimates of .course lengthy 

The statistics w> the goodness of fit of the regression of the trajectory model parameters on the ,^ 
basehne parameters are presented in Table 14. Judging from the extremely low values of the square of 
multiple.conelation coefficii^ts, .11 and .1 3, and the large standard enors of estimation, the fit appears 

. rather poor for the estimate oiz. The estimate ofjb is a bit better, with the "square of the multiple conela- 
tions of .26 at the block level and .18 at the lesson level-. How the adequacy of fit of;a_and_b_ translates into 
the adequa'cy of fit on cumulative times will be discussed later in the comparison of models. The first three 
variables to enter the estimation of the b.coefficient - the total score on the IM/MF reading skills test . . ■ 
(RE'ADST), sex (SEX), and the score on the experience with conventional instruction scale (EXPCI) - 
entered first in the baseline mUestone model regression's as weU. The trajectory model thus appears to pro- 
vide a good descriptive fit of the data and leads to prediction of process on the basis of the same baseline 

• variables as a^re used in the milestone model. 

* 

Results for BTM2 

:•. . The BTM2 model, expressed by the equation tf = aj + bjA, was evaluated only at the block level for. 
•reasons that will be discussed below. The parameters in this model were estimated by finding the'^and b_ 
coefficients for sefected^alues of IO(K ranging from .3 to L9 in increments of .1). The value of K^tiiat 
minimized the sum c/squares of the residuals was taken as the estimate of theXT)arameter and the 
associated^and^estmiates were assigned the corresponding values- , 
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Table 12 

GOODNESS OF HT STATISTICS FOR THE LINEAR TRAJECTORY C 
'MODEL (BTMI),AT THE BLOCK AND LESSON LEVEL 



Block 



\ 





1 




3 


4 


5 


6 


DiOCJC LjCVci 














RcsidusI st2tti3tics 














Me^n^ - ^- 






■so 








Standard deviation 


111 ^ 


1 TO 


11 1 


234 


I5o 


243 


"Median' 


'34 


-42 


-9 


-1 


24 


• -21 


Mean absolute 


163 


128 


195 


175 


120 


186 


Root mean square, 

1 


* 221 ■ 


183 


274 


234 


160 


245 „ 


Lesson Level 














Residual statistics 














- Mean 


-10 


24 


12 


-12' 


-10 ' 


'"• -13 


Standard deviation 


134 


176 


'21,7. 


174 


148 


307 


•. Medial}^ 


-10 


21 


■17 


0 


-15 


:rll 


Mean absolute 


103 


_13.4 


167 


133 


' 117 


. 242 


Root mean square 


134 


111 


217 


175 


, 149 


307 ^ 








Table 13 









MEAN AJ^D STANDARD DEVIATION OF Tlffi^STIMATED PARAMETERS^OR 
• THE BASELINE TRAJECTORY MODEL 1 

•■ (n - 368) . ' . ■ 



Block Level 
/\ 

a ■, 
b ■ 

Lesson Level 
a 

/s 

b 



Mean 

-121 
96 

5*0.7 , 
72. 



SD. 

464 
23 

297 
17 



/ ■ 
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/ /Ttie dStrSbution of students on the value W K that minimizes the residual sum of squares is given in 
Kigare 1 . The riode of the distribution is at K =\.0. Of aU the students, 22% had a.yalue of K = 1 .0 that 
iniriimized the tesidual sum of squares and 77% nid values between .8 and 1 .2, inclusive. 

j^iecaiise/of the way in which the achievemAntMndex was. defined, it is not surprising that most of the,, 
values oflTduster aboutM. The achievement indfexlvas based on a standardization of mean lesson times so 
thftt each lesson was assigned an achievement vaRie eiuatto the percentage of cpurse time spent on the 
lesson. The fact that the optimum K value was/apprc^ipa^ 1 for most students indicates that the pro- 
-portion of time spent on a particular part op<he coa^ to be stable across students. 

As was found iii another examinati^rK^t^ trajectory model (Larsen, Markosian, and Suppes, 1977)^ 
\yhen Kis estimated' individu^lx_bjfjurde values of K, a, and b are highly dependent. For example, 
•^tablel^-pre^ estimated b coefficient as a function of K. The range in the mean 

vdugjafliie estima^^ the b coefficient i/from J for K = .5 -to 477766 for K = 1 .9. This dependence may 
"^e^plained by the w^(iir^5ch,K tpX^ tHe model- as an exponent of time. For a given observed course 
length, the value of the a and b coefficients would be expected to increase as K increases* 

' Table 16 presents the summary statistics on the descriptive fit of the nonlinear trajectory model. The 
low values of the mean and median indicate that^the bias in the model is negligible. The values of the mean 
absolute residuals and root mean square residuals indicate a good fit to the d^ta. Comparison of the 
statistics in Table 16 with those in Table 12 show that the nonlinear model improves the descriptive fit - 
substantially for the first and last blocks. « " 
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STA-mXICS FROM THE REGRESSION OF THE LINEAR 
TRAJECTORY PARAMETERS ON THE BASELINE VARIABLI^S 
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Dependent Nimble 

with all baseline 
variables 

r2 with/truncated set 

Standard error of 
estimation 

Variable entered (r2) 
-Step 1 



3 
4 



^ 8 
9 
10 
11 



Block Level 



.11, 
.10 

. ' A 

'445 ^ 

READST (.03). 
^ SEX (.06) 
MEMNB ^ (.08) ^ 
LOGREA (.09) 
ENTAGEf (.09) 
STCUR (.10) 
RVOCX3N (.10) 



READST 
EXPCI 
SEX 



lESCL 



,^ TRCUR 
^IIEADST 
HIYEAR 
ENTAGE 



.26 
.26 

' 20 

, (.11) 
(.16) 
(.21) 
;(.22) 
(.23) 
(.24) 
('24X 
(.25) 



■ .13 
.11 

282 

RVOCTL (.07)^ 
HIYEAR . (.08) 
STCUR (.09) 
EXPCL • (.10) 
SEX (.M) 
tRCUR (.11) 



STCUR C25) 
MEMNB (.25) - 
TSTANX (.26) , 



Lesson Level 



.20 . 
' " .18 

16 . 

READST (.08) 
EXPCI (.12) 
, SEX (.16) ' ■ 
^ TRCUR , (.17) 
LOGREA (.18) ' 
CONFIG" : (.18) 



- 28 - 

. 30 
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Table 15 — .— 

STATISTICS ON THE ESTIMATE OF THE- 
COEFFICIENT AS A FUNCTION OF K 
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JL 


Mean 


C T\ 
b.U. 






.5 


.7 


.06 


2 




.6' 


- 1.9 


" . .35 * 


17 


4 


.7 
•8 


. 5.0 


. .89 


27 


. \ 


'13.5- 


2.24 ' 


.43 




• .9 


35.7 


7.09 


04 




1.0 


94.8 


22.45 


79 


1 


1.1 


.257.2 ' 


66.93 ' 


' 60 




1.2 


721.7 


. - .210.07 


38 




1.3 


1767.9 


436.88 


17 




1.4 


4881.0. 


1104.23 


9- 




- 1.5 


. 10723.9 


4597.95' 


■ 6 


/ 




38168.0' 


17598.96 


4 


t 


1.7 


89632.0 




1 




1.8 


477766.2 • 




1 



* ■ 

Table 16 

GOOdNESS OF FIT STATISTICS FOR THE NONLINEAR 
- TRAJECTORY MODEL (BTM2).AT THE BLOCK LEVEL ■ 

Block 

_1_ ' _2_ _3_ \4_ J. A. 

• Residual Statistics ^ . 



Mean ' 


.20 


-62 


35; . 


12 


39 


-45 ■ 


Standard deviation 


74 . 


195 


146 


152 


154 


116 


Median 


' 10 


-29 


18,' 


:^..:,3.. 


V 31 




Mean absolute 


"57 


139 


103 


116" 


. 118 


95 


Root mean square ' , 


76 '. 


205 


150 


, 153 


158 


124 
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Estimation of K, a, and b using Ithe baseline variable^^has presented major problems. Because. the 
>alue of K corresponds to the exponent of the dependent variable, estimation of the three parameters * 
" separately would not be fruitful. 

One alternative that was.eXplored was-to attempt to predict K using the baseline variables ands then 
. \ to estimate a and b as a function of K. As a first step, K was entered as the dependent variable in a stepwise 

regression with the baseline variables as independent variables. 

. •.* ■ 

The results of the regression, however, indicated that th^ baseline variables were poor predictors of 
K. With all the variables entered in the regression, the^ square oF the multiple^orrelation coefficient was 
only '.13 and the standard error. of^stimation was .2. With such poor prediction of K and the sensitivity of 
the nonlinear model to the value of K, the nonlinear model does not appear to.be useful for predicting 
student progress. Augmenting the trafktofy model with a nonlinear component apparently improves 
the model fihmlhe point of view of describing the data base, but makes the model too sensitive f® pre^ 
dictive purposes. ; 

Results for PTM 

Table 17 presents the means and standard deviations of the parameter estimates for the baseline 
trajectory model at the end of each block. The estimates at the end of block 1 are left blank in the block 
level analyses because the PTM model requires at least two points of observation. The model, it may be 
recalled, is: 

t = a + bA, 

where t is cumulative time, A is cumulative achievement, and a and b are parameters to'be estimated. For 
' the block level analysis, the mean of the estimates of a are consistently negative, but with a very large stand- ' 

ard deviation. For the lesson data, the mean of the estimates of a^are substantially closer to zero, with much 
• smaller standard deviations. The lowerstandard deviation of the estimates of a at the lesson level is probably 

due to the larger number of data points that enter into the estimation. For example, at block 2 there are 2 

data points for each student at the block level and 19 data points for each student at the lesson level. . r ^ 

^ Another point that may be noteworthy about Table 1 7 is that the standard deviation in the estimates 
of .b decreases substantially between the initial block and block 5. At the block level, the decrease is from 
30 niinutes^to 24 minutes; at the lesson level the decrease is from 25 min 
5^^. ct)ntini^^^ when the.Blpck: 64ata are included in tJie estimation. Tlus decline ^ , 

: probably due in part to the increase in the nuinbif^ of data points acrossrtlocii: It may alsb indicate that ' ^ 
the learning rate is becoming more homqgenepus over time. Note, however, that the parameter related to - ■ 
. >fe3xning.rate, b/stiU has a^ather hrge standard deviati^^ 

. Results for MMM- * ^/ ■ 'J'- ' • ..\ ■ ' .y'"' . ' . • 

Table 18 contains the summary statistics for the regression of remaining cumulative course time on ^ 
both the performance and the baseline variables. Comparison of these data with those presented in ' 
Table 1 0 for the performance milestone model indicates that the baseline variables add v^ry little in . 
predicting course completiori time in addition to what is explained by the performance variables. At. the 
- block level, the standard error of estimation decreases by about 100 minutes as the result of including the 
: baseline variables at the end of Block 1. In all other cases, the improvement in prediction is^ negligible. V 
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Block 



Lesson 









a 


b 




/s 
a 




b 








Mean 


S.D. 


Mean 


S.D. 


Mean 


S.D. 


Mean 


SX). 


At the end 




















of Block: 


1 










-4 . '> 


69 


75 


25 ■ 




2 


-64 


386 ■ 


92 


30. 


-1 


101 


■ 74 


22 




3 


-140' 


413' 


96 


28 


17. 


159 




20 




4 


■ -133 


. 410 ; 


96 


26 


31 


202 


72 


19 




5 


-140 


428 


96 


24 


45 


257 


72 


18 



: ■ Table 18 . 
STATISTICS ON THE REGRESSION FOR THE MIXED MILESTONE MODEL 



Block Level 
Block 1 
2 

. . 3 . 

u 

- 4 
. 5- 



Truncated Run 
Number of Base- 
line Variables 
Entered 



6 
1 
0 
0 
0 



r 

R2 on 
Truncated Run 



.57 

.75; 



.Standard Error 
' ' ''of. 
Estimation 



1449 
1105 



Lesson Level 
• Block \ 

2 ■ ■ 
3 
4 
5 



,4 
0 
0 
0 
0 



.53 



1137 



Note: indicates no baseline variables entered. 
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Comparison of the^ Baseline Modds - — 

The goodness of fit statistics for the three baseline modek are arrayed in Table 19 for the block level 
analysis an* in Table 20 for the lesson level analysis. The goodne.ss of fit statistics indicate that all three 
models fit the data to about the same degree. The mean residuals are uniformly^mall, at both the block 
and the lesson levels. The median residuals are consistently negative and consistently exceed 100 minutes 
for Blocks 3 through Relative to the magnitude of the mean absolute residuals and the root mean square 
residuals, however, the medians appear relatively small; and the bias m the models does not appear to be 
large enough to be a factor in model selection. * 

The mean absolute deviations and the root mean square deviations increase substantially over blocks 
for all models. These statistics are about five times larger at\he end of the sixth block than at the end of 
the first block. Of course, this is a directreflection of the relative magnitude of the cumulative time to the 
end of the course versus'the cumulative time to the end of the first block- 

Comparison of the Performance Models / 

The predictive accuracy statistics for the two performance models are arrayed in Table 21 for the 
block level and lesson level analyses. The statistics arcvfor the distribution of observed minus predicted 
course time using baseline data and time-to-criterion data up to the end of each block. There is no entry 
for Block 1 for the trajectory model at the Block level because this model requires at least two observations - 
for purposes of estimation. . ^ 

' J The predictive accuracy statistics indicate that the performance milestone model is better at predicting 
coilrse time than the uajectory model, both at the block and lesson levels. For all blocks, the mean 
absolute residual and the root mean square residual are much higher for the trajectory model than for the 
milestone model. In fact, the size of these statistics indicates a lag of about one block between the, two 
models. That is, it takes about one more block for the trajectory model to meet the accuracy of the mile- • / 
stone model - 

The performance trajectory model also has a rather large bias in the first few blocks. At the block 
level, the large positive values of the mean and median residuals at the end of Block 2 indicate a positive 
bias, which means that the model is predicting too low a value for the remaining course time. At the 
lesson level, the large negative values for the mean and median residuals at the end of Block 1 and Block 2 . 
indicate a negative bias, where the model is predicting too high a value for the course time. Also, it is 
interesting that at the lesson level the mean absolute residual and root ihean square residiial are larger for 
the performance trajectory mgdel at the end of Block 1 than for the baseline trajectory model. 

• One explanation for the difference in predictive accuracy betv/een the trajectory and milestone models 
is that the milestone model tends to rely heavily on the most recently observed cumulative time, whereas 
the trajectory model tends to' rely on aU the obse^ Thus, the lag in the predictive . . . ; 

accuracy statistics for the trajectory model may^he attributed to the'weighting that : this tnodel gives.to early, 
cumulative times. In any event, the performaiice milestone model appears to be superior, to" the perfprraahce 
trajectory model, at both the block and the lesson levels. , ^ ' - 
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„ . _ .TableJ9 _ 

GOODNESS OF FIT STA'nSTICS't?( 
BASELINE MODELS: BLOCK 



[HE THREE 
JALYSIS 





1 


' 2 . ' 


. V 3 ■ 








. \ BMMl 








• .\ ' . . 
.' 


. , • \ ^ 




Mean 


..0 


0 • 


' . .0 


■ V ' 0 


0 ■ 

• » 


. .0 


Standard deviation 


• 384 


734'. 


. '1216 


1448 


•1717 


. 1865 


Median 


-39' 


-80 


,-28 


• .-135 • ' 


-95' 


-117 


Mean absolute 


280 


556 


952 


1146 


1368 

» * 


1480 


Root mean square 


384 


733 , 


: 1214- 


■ 1446 ■ 


, 1715 
i •■ 


1862 


BMM2 


• 




• 








Mean 


— 


— 




—- 


• — , ■ 


• 0 


Stiandard deviation 


397 


- 746 , 


, 1227 


1454 


1720 


i865 


— Median- 


-47 


-79 


-114 


-138 


-151 


-,-1-17 


Mean absolute 


290 


- 575 


963 


1155 


1'372 


1480 


Root mean square 


396 


■ 745 


1225 


1452 

n ■ 


1718 

f 


1862 


• • - ' ■ 
' BTM 














Mean. 


13 


-45 


32 




34 


-36 


■ *^ : Standard deviation 


393 


735 


1217 


1446 


1714 


1865 


Median 


-L6 


-96. 


1" 


-136 


' -72 ■ 


' -83 


Mean absolute 


S.283 


561 


949 


J 146 


1365 


. 1477 


Root mean square 


. '^392 


735 


1215 


1444 


1712 


1863 

















Note: "-" indicates between -.5 and +.5, but not identical to zero. " 
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. , Table 20 _ _ , 

GOODNESS OF FIT STATISTICS' FOR THE THREE BASELINE MODELS: 
. LESSON LEVEL ANALYSIS; AT THE END OF EACH BLOCK 

^ 1 Block - 





\ BMI O 


1 


1 

1 ^ 


_3. 


_4. 


_5_ 


6 


{ 
















Mean 


0 


0 


0 


0 


0 


0 




Standard deviation 


283 


574 • 


918 


1136 - 


1320 


1450 




Median 


-32 


-75 


-108 


-166 


-121 


-131 




Mean absolute 


220 


450 


735 


.927 


1069 


H78 




Root mean square 


283 


573 


917 


1135 


1318 . 


1448 




BMM2 








• 








Mean 


35 ■ 


63 


42 


12 


7 


0 




Standard deviation 


295 


589 


"932 


1143 


1322 


1450 




' , Median 


4 


6 


-83 


-201 


-142 


-131 




Mean absolute 


234 


470 


751 


942 


1071 


1178 




Root mean square 


297 


592 ^ 


932 


1142 


1320 ' 


1448 




•BTM 
















Mean 


-10 


24 


■ 12 


-11 


-10 


-13 




Standard deviation 


286 


. 582 


928 


1142 - 


1327 


1461 




Median 


-37 




-122 


. -239 


-165 


^150 




Mean absolute 


. 22i 


458 


747 


'937 


1070 


.1187. 




Root mean square 


286 


582 


927 


. 1140 


1325 


1459 



The mean absolute deviations and the root mean square deviations increase substantially over blocks for all 
models. These statistics are about five, times larger at the end of the sixth block than at the end of the first 
block. Of course, this is a direct reflection of the^ relative miagnitude of the cumulative time to the end of 
the course versus the -cumulative time to the. end of the first block.- 



EKLC 



-34 



56 



AFHRL-TR-78;7 



AFHRL-TR-78-7 



Table 21 

GOODNESS OF FIT STATISTICS FOR THE TWO 
PERFORMANCE MODELS: BLOCK LEVEL AND LESSON LEVEL 



Level, Model, and 
Residual'Statistics 


Baseline 


Block 1 


Block 2 


Block 3 


Block 4 . 


Block 


Block 














Milestone 














Mean 


0 


0 


0 


0 


0 


0 


Standard deviation 


1865 


-1552 ■ 


1113 


728 


532 


292 


. Median 


-117 


-105 


-68 


-59 


-20 ^ 


-29 


iViean aosoiuie 


1*tOv 


1910 


o / ^ 






230 


Root mean square 


1862 


1550 


1112 


727 


532 


292 


Trajectory 

4 

Mean j 


-36 . 


■ 


222 


-75 


• 

-54 


-68 


Standard deviati(»i . 


i865 




, 1529 


969 


728 ^ 


458 


Median / 


-83 


" — 


316 . 


29 


8 - 


-40 


Meai) absolute 
ivool incaii squoTc 


1477 

A. 0\J^ 


** 


1134 


710 
970 


543 

729 


351 
. 462 


Lesson 














Milestone 














Mean 


0 


0 




0 


0 


0 


^ Standard deviation 


1460 


■ 1146 


938 


■ 643 


470 


251 


Median 


-131 


2 


16 ■ 


-71 


-37 


-21 


Mean absolute 


1178 


915 


768 


. 514 • 


'375 


196 


Root meaa square 


1448 • 


1144 


937 ' 


■ 642 


^ 470 


251 


Trajectory 
Mean 


-13 


. -280 


-232 


-124 ' 


-66 


-26 


Standard deviation 


1461 


. 1823 


1297 


907 


691 . 


477 


. Median 


-150 


-204 


-260 


-111 


-95 


^32 


Mean absolute 


1187 


1411 


1042 


,732 


554 


.378 


> Root' mean square 


1459 


;i842 


1316 


914 


693 - 


.477 



Note: Each column contains the goodness of fit statistics for reinaining course time using the cimiulative 
performance data throu^ the specified block. 
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IV CONCLUSIONS AND RECOMMENDATIONS 

Conclusions Based on the Results of the Evaluation 

Based on the results of the Uterature review summarized in Appendix A, trajectory and milestone 
models of student progress were formulated and evaluated. The evaluation, using block elapsed times and^ 
lesson times to criterion, was conducted separately on 368 students who had completed the Inventory 
Management Course early in 1977. Four models were evaluated using baseline data only; two models Vere 
evaluated using performance data only; and one model was evaluated using both baseline and performance 
data. V ' , / 

Although the nonlinear trajectory model provided the best descriptive fit to the data, it was found 
to be a poor predictive model because of difficulties in prediction of the exponential parameter K. None 
of the other three baseline models appears to be substantially superior to the other two on the basis of the 
predictive goodness of fit statistics. In particular, the BMM2^model, which corresponds to the model used 
in the currently implemented SPMS, was' comparable in accuracy of prediction to the other two models. 

For the performance models, the milestone model was substantially better than the trajectory model 
in predicting course completion times. The trajectory model appears to need performance data on an 
additional block to achieve the degree of accuracy in prediction displayed by the milestone model. 

In comparing the results for the baseline milestone model with those for the performance milestone 
modeU it is evident that prediction of student progress can be made more accurately from initial perform- • 
ance data than from baseline data. This result is consistent with whafwas found in several other studies 
included in the Hterature review (Wagner et al., 1973; Yeager and Kissel, 1969; Wang, 1968), namely, that 
the best predictors of student progress are those that are most related to the coyrse cbnteuL A measure 
of actual performance on an initial segment of a course, then, will be a good predictor of student progress 

if the course is relatively homogeneous in the the types of skills that are necessary for learning the contents. 

' ■ • * , ■ ^ 

For the Inventory Management Course, the accuracy of prediction of course ^pompletion time can be 
improved by using initial performance data. For example, when the block elapsed time on the first block 
is used to predict course completion time, the standard error of estimation is 334 minutes less than the 
standard enor of estimation using the baseline data.'Using the first two block elapsed times as predictors, 
the standard error of estimation is 772 minutes less than that derived using the baseline data (see Table 10). 
Of course, the amount of time remaining in the course is also decreasing at the end of successive blocks. 
Nevertheless, the increase in precision of prediction of course completion time is appreciable.. . 

Finally, on the basis of the results for the mixed milestone model, it ma'y be concluded that aug- 
menting the performance data with baseline variables as predictors modestly improves the precision of 
prediction at the end of the first block. However, at the end of subsequent blocks, the baseline variables 
do not add substantially to the precision of prediction obtained using performance data alone. 

Other Considerations 

As was indicated in Section I, the AIS currently has a student progress management component 
that was implemented in July 1 977. In considering recommendations for Phase III of this project, it is 
valuable to review some of the initial results found for the currently implemented system, as reported .by 
Dallman and Grau (1977). Comparisons of performance data between students who were in the IM - 
course before the implementation of the SPMS with students w^o were in the course after the SPMS was 
-implemented indicated: 
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• The average block elapsed times for the SPMS students were about 10% less* than the 

: average blockrelapsed times forthe iion-SPMS students; — — ; 

• There was a definite decrease in block grades and a definite increase in first-attempt 

failure rates when the SPMS was implemented. , . 

Dallman and Grau's recommendations focus on what course managers, su^pervisors, and instructors 
can do to improve the management system. Apparently, the procedures for generating the target completion 
times» including the prediction equations, were found to be satisfactory. 

Recommendation for Phase in - • . > ' 

Our recommendation for Phase III is to implement a form of the performance milestone model on 
the AIS to augment the currently implemented baseline model. The currently implemented model provided 
about as good a predictive fit as the other baseline models that were examined. Therefore, there is no 
reason to modify the procedure for predicting student performance based on the baseline data alone. 

Our evaluation confirms what many researchers have found (Wagner et al., 1973; Wang, 1968; Yeager 
and Kissel, 1969), namely, that the precision o?^rediction of student performance increases as a function . 
of the relevance of the predictor variable to the course content. By taking initial performance measures on 
the course itself, it would be possible to improve prediction of performance on the remaining course 
material, if the material is reasonably homogeneous throughout the coursjj. • " 

Within the currently specified level of effort, Bhase III would consist of the following tasks: 

(1) Familiarization with the coding aSd desf^ of the AlS, with special attention paid to 
- the Adaptive Model and the student progress management system. 

(2) Consultation with AFHRL and other persoimel at Lowry regarding implementation of 

- the model. , ' . 

(3) Formulation of specifications for integrating the performance model into the existing 
AIS, resulting in a detailed integration and design document. . • . , 

(4) Implementation of the modifications using CAMIL and testing for reliability, 

(5) Briefing 6f course personnel on model output. ^ 

- (6) Evaluation of the model. ' ' . 

(7) Preparation of the final reports 

The performance model as implemented could be utilized in several differertt ways. At a minimum, 
updated predictions could be used in the scheduling of the administrative outprocessing activities and 
^^rogress counseling sessions. ^ ^ ' 

The updated predictions could also be used in a'modification of ^he current student jjianagement 
system. For example,- rather than students being provided with their targets for the entire course, they 
could be informed in a- stepwise fashion- They would be given their target times on the first block or the - 
first few blocks based on the baseline model,. The targets on" each subsequent block could" then be provided 
on the completion of the prior block using the performance model predictions- ^ 

Under this scheme, provisions would need to be made to preclude increasing target times for ^ 
- students not working up to their capacity- For example, predictions could be set as the lower of those 
generated from preassessment data and those generated from performancf data. Or they could be 
established as a weighted average of the two predictions. \ , 
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An alternative use for the performance modpl w6uW be in the determination of when progress 
counseling sessions are necessary , The current SPMS specifics progress counseUng sessions in the cvent^that- 
a student falls two days behind his tarjgeted path through the course. This could be replaced by a cjiferion 
that is a function of the difference between a 5tu4cnt*s targeted completion date based on prcassessment 
data, and a student's projected completion datc'^ased on his performance. This type of criterion could be 
more sensitive to lags in. performance at the beginning of the course. For example, a progressxounseling 
criterion of a difference of two days between the baseline model and performance model predictions of 
course completion time would translate into a difference of substantially less than two days between 
observed and predicted times on the first or second'blocks. This approach would be equivalent to setting 
criteria for differences in learning jates rather than differences in learning time. This approach could lead 
to earlier detection of students in need of progress counseling and remedial iiistruction. 
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, _ \ Appendix- A/- ■•J:;'--- ''. ■ ■ v 
^ mtRATURE REVIEW' ■ , , 

The Kterature review covered two areas relevant to the management of student progress: (1) inathe- 
noatical and conceptual' models of student progress,' and (2) incentives and intervention strategies to help 
improve student motivation or study niethods. The emphasis was on the review of models of student prog- 
ress since the pxiiijary thrust oFthe. project was on^he formulation, evaluation, and ultimate implemes^ation 
of such a model. The review of interventfon strategies was inteiided to provide. insight into how tEe ifiodel 
of student process could be used in the context of the Advanced In : 

Description of the literature Search - , _ _ . , 

■ ■ ■ ■ . *■ ' , . "• . 

The literature search used both computational and manual methods to identify rjeferences that were • 

potentially relevant to the project. Computerized searches were made through the following files: 
. ' .. • Smithsonian Science Information Exchange (SSIE) — current projects by definition. 



• 



.Resources in Education/Current Index to Journals in Education (ERIC) — 1^66 through 
May 1977. ^ ' ,. ' " / 



• Psychological Abstracts --r 1 967 through April 1 977. 

- * ' • Computerand 'Gontrol Abstracts (INSPEC) - 1970 tfo^ - ' 1 

• . National Technical Information Service 

: • Quarteriy Bibliography of Computers and Data Processing — 1968 through 1976;- - 

• Technical Abstract BuUetin - 1969 through May 197^^^ , . ' ^ ^ 

. . The search strategy was designed to match" the concepts of computer-assisted and computer-managed 
instruction with the concepts of adaptive contrpl, optimization, mathematical models, and related terms. 
An attempt -was made to restrict citations for applications of computers to instruction in the military and 
te'dmical domains; but this proved to be too limiting. Therefore, the domain of instruction was not used - 
.as a delimiter in the computer search in order to include as many potentially-relevant sourceis as possible. / 

Th^ manual search included examination of recent issues of selected journals, r^$few of bibliographies \ 
in references already identified, identification of references from conversations with experts in thefield, 
and reqxieists rnade to Hbrariaiis supervising specialized coUectiom. : / , . / / 

The titles and available abstracts of all the references produced from both the computerized and ^. 
manual searches were scaiined for relevance. From thousands of references, about 400 remained after the. / 
^initial scan. About 40 6f these were then selected a!s thfe ir^relevatnt'for this particular project. These 40, v 
plus the references listed in the.Jlequest for Proposal, were. thenTreyiewed^iri detail it^ preparation for the • 
literature review that follows. The review is organized into four sections: perspective on individualized 
instruction, models oiF student progress, incentives and intervention strategies, and conclusion. 
. - * ' . .... 

Perspective on Individualized Instruction 

.' Jhe Advanced Instructional System at Lowry AFB is tesed on a strategy of individualization of - ' 
instruction. Cooley and Glaser (1969) define individualized education as "adapting instructional practices : 
to individual requirements.'* In the case of-the Advanced Instructional System, individualization is C - . 
achieved by the self-pacing of instruction and by providing alternative modes of instruction on selected 
lessons. . ' > , • * . 
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Gibbons (1970) traces the development of individualized instruction to before the turn of the' 
century, where it ori^nated primarily as a reaction to the conventional age-graded eduj^atiqnal system, 
Sass (1971) cites the wofkof Washburne (1922) as an early attempt at individualization of instruction, 
Washbume's idea was to reverse the constraints on time and achievement in an educationaTprogram. 
lUther than having fixed blocks of time for instruction 

advocated that an^ducatibnal program be designed so that all students attain a specified level of achieve- 
ment. In more rec^t. years, .Carroli (1963) and Bloom (196^^ 1976) have extended and refined this 
approach in their work on mastery learning. ^ 

. At present, individualization of instruction is one of the most dominant concepts in educational* 
innovation and reform. This emphasis is due in part to the increasing application of computer technology^ 
to education, which has-made it possible to process the larg^ amount of data necessary to truly mdividuali^ ' 
instruction and provide immediate feedback andguidance to the student (Suppes, 19frt; Cooley and Glaser, 

1969) . '/ ., 

As the concept of self-paced instruction has been increasingly accepted, much research has gone into 
examining rat^ of progress. Some of this work MT^een directed toward determining what factors are 
related to rate of progress in a curriculum (WangS^SeS, 1970; Yeager and Kissel, 1969; Wang and lindvall, 

1970) . Related work has been concerned with modeling and predicting rate of progress (Wagner, Behringer, 
and Pattie, 1973; Suppes, Retcher, and.Zanotti, 1975, 1976; Malone et al., l977)U)ther work has been 
concerned with alternative strategies for optimizing rate of progress {Anderson, 19x6; Wang, 1976). 

In the next section, we tarn to describing the models of student progress that htfw been developed 
and summarizing their descriptive and predictive adequacy. We then bfiefly review some of the most 
relevant literature on incentives arid intervention strategies." 

»• - - 

Review of Models of Student Progress 

The references regarding models of student progress may generally be categorized by their degree of ■ 
specificity. A number of studies have taken a conceptual approach to the development 'of models of student 
progress/the object being the development of a framework for a theory of instrucOon, In other cases, the 
authors specify axioms^ or assumptions regarding the processing of information and derive specific functional 
forms for the relationship between achievement andelapsed time. In the former category are included work 
by CarrolL(l 963),. Bloom (1976). and Cooley and Lohnes (1976-). In the latter category are included work by 
Suppes and his associates (1975, 1976), Chant and Luier^berger (1974), and Hicklin (1976). 

A distinction that has been made by Suppes, Macken, and'Zanotti (J977) is that between studies at 
the microscopic level, concerning learning of specific types of material under specific types of reinforcement 
schedules, and studies at the global level, \^*iere the focus shifts from protocols of responses on specific 
trials to mean performance over substantial periods of tiine. The psychological literature is replete ^with ' 
studies at the ihicroscopic level of detail. However, these studies are of little use in the eiideavor .to develop 
a model of. student progress appropriate t<^he AIS, therefore, only the models at the global level that 
would appear to have some use in the current project are included in the literature review. * * 

The three mathematical models are first described; this is followed by a description of several con- 
ceptual models; firially, the results of prior evaluations of mathematical models are "described. 

MathematicalMpdels of Student Progress . ; , _ 

. Each of the three global modek. of student progress are based on assumptions regarding how students 
prdcess information. The three models are: the trajectory model (Suppes et al:, 1976), the dynamic 
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equilibrium model (Kclclin, 1976), and the generalized Thurstone^ model (Chant and Luenberger, 1974). - 
The Trajectory Model ^ ' . ; 

Suppcsf Fletcher, and Zanotti (1976) begin with five assumptions regarding processing of informatiori. 

Let y(t) = the poation of the student in the course and y (t) = the student's rate of progress through 

through the course; A(t) = cumulative amount of information introduced in the course up to time and 
. A(t) = the rate of introduction of information; and s(t) = the student's rate of processing or sampling 
Wormation- The ifive axionis may then be expressed a^ 

Axiom 1 : s(t) = k j A (t)/A<t) for some constant k 1 . 

Axiom 2: Upon introduction of a new piece of information at time t, for a small 

intervalof time h, s(t+h) = s(t)- [s(t)--s(«)]s(t). . 

* • '■ ' ." ■ ' ^ ' - ' ■ , . ■ ■ , , ■ ' ■ 

Axiom 3: . The probability that a new piece of information is introduced for a given 

student al-time t is independent of t and the previous introduction of ^ . 
V . . information. ... 

Axiom 4: y(t) = k2A(t) for some constant k2. ' ^ 

Axiom 5: y(t) = k3A(t) for some constant ks- , 

In their discusdon of the axioms, Suppes et aL state that ^they are least satisfi^ 
because of "the absence of a more fundamental qualitatiye characterization of the rate assumption 
expressed in this axiom/' T^ey felt that the other four a^qp^ms have a "natural intuitive content that does 
not require explicit discussion." . , 

The basic equation for the trajectory model, ^ . 

, y(t) = bt^ + c 

was derived. from the five axioms. It was stressed that this relationship between course position and elapsed 
time was stochastic rather than deterministic; that is, it represented .what would occur on the average for a 
•given,student.:The parameters b, c, and k were meant to be estimated separately by student rather than as 
a function of group data: ' 

' The Dynamic EquilibTjium Model : ' ' 

. HBcklin V(l 976) dynamie-equaibriummodels are based on the assumption that at any particular, time 
't there are TnI units of niaterial* Let * 

^1(1)= the amount of material yet to be assiinilated 

N2(t) = the statiis of the individual « 

N 3(1)- the amount of material in the lost category. . 

. « .■ • . '1 . * • ' > 

'nien,N=N,(t) + N2(t) + N3(t). .. . 
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His basic assumption is that during a time interval At^ N2(t) will iiicrease in proportion to the amount of 
material to be assimilated, Ni (t). and will decrease in jproportion to the student's current status, N2(t). 
lAider these assumptions and the initial condition that N2(0) = 0, the "basic differential equation of 
dynamic equilibrium theory'' can be derived: ' 

dN2(t) _ j^^(t) - ksNjCt) for some constants ki and . ' 

V'-. ' • dt - ..■ . ' ■ • ■ - ^ 

Under alternative assumptions regarding the values of ki and k2 and what happens to lost material, 
Hckljn derives three models: . • . 

Case I: Under the assumption that k2 - 0, NaCO) = 0, and N2(0) = 0, . ^ 

N2(t) = N(l-e-^20- ^ * • 

Case H: Under the assumption that lost material reverts to the unassimilated 

state, k2 =?^=0, and N2(0) = 0, ' / 



Case in: Under the assumption that k2 0 and N2(0) - 0, 



^ Generalized Jhunton^Model 

The model of Instruction/Learner Interaction proposed by Chant and Luen^>erger (1974) was a 
generalization of Thurstone's model. Thurstone's model (1930) was based on assumptions regarding th 
relationship of the state of the learner^ p(t), and the number of potential successful acts, s(t), and fail' 
, acts, e(t), in the learner's repertoire at .time t. The function p(r) represents the fraction of total learn 
and by definition: 



:.,.._^s(t)__ 



'Assuming that 



ae(t) 

■ . ' ' m. 



= -k[l -p(t)J.and 

= s(t) e(t), where m is a constant. 
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ithe basic differential eqiiation of ThurstoneV model can * . 

. : . V - - - 

The function p(t) is asymptotic to p = 6 as t decreases and to p = 1 as t increases. It is symmetric about 

■ p = l/2, .. ■ ;;. \ . ■ \. ■ ■■ j ... ■ ■ • 

Chant and Luenberger generalize the Thurstone model by proposing the following differential 
equation to specify the :5tate of tlie learner: . / , 

■ = u(0 ag[p(t)] . / -■ , • . .. 

where the function g is assumed to be continuous and to approach zero as its argument approaches zero " 
or one. The function u(t) is called the instructional input variable and is assumed to represent the effect of 
instruction. The function g(p) is -called thte characteristic learning function and represents characteristics of 
the learner arid the material to be learned. The variable a in the formula is constant for each individual and 
is intended to represent the student's aptitude. 



Conceptual Models of Student Progress / , . .. * 

.In contrast to the detailed mathematical models described in the previous section, the conceptual 
models, provide schemes for examining the variables relevant to school Teaming. As such, they incorporate 
more factors, than <io the mathematical models, but they lack the specificity of the mathematical models. 
The relevance of the conceptuial models.to the current study is in providing perspective regarding the role 
of "lime to learn" in past researcTi. Wagner et al. '(1 973) attribute the use of "time to learn" as a critical ^ 
variable in modem educational and training research to a model of school learning developed by John 
Carroll (1963). According to Carfoirs model, the degree of learning a given task is a function of the amount 
of time spent on learning the task and the amount of time needed to learn the task. Thus Garroirs model* 
embodies Washbume's idea of regarding time as a variable ratherthan as a given qiiantity. "Time spent," 
iji CarroilVsch^me, depends on opportunity, perseverance, and aptitude; "time needed'\depends on ■ 
aptitude, ability of the student to understand instmction, and quality of instructicm. 

"Opportunity" and "quality of instmction" are attributes of the educational environment. The 
former is measured by the amount of time allowed for learning; the latter is defined with respect to the 
efficiency of instruction and is asseSsed by the degree to which the amount of'time needed to learn is 
minimized. . 

"Perseverance," "aptitude," and "ability to understand instmction" are attributes of Ae individual; 
student. "Perseverance" is related to a student's willingness to spend the time necessary for learning the ' 
. task; "aptitude ," in Ca^oll's scheme ,.is.deflned:as:"the amount of time needed to learn the task under 
optimal instructional conditions"; and;"ability to understand instmction" was considered to be a factor 
dependent on general intelligence and verbal ability. . . ' 

The-iiQportance of Carroll's model was in regarding time as a major ^^riable iri. predicting the degree 
of learning. Thus aptitude was defined with regard to the tinie necessary to' master a task rather than the 
leveLof mastery within a given tinie. • 
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'.- ' ^ ; • ^ • . . 

Carroll's model-has served as a basis for paradigms developed by Cboley and Lohnes (1976) and by 
Blo6m (1976). The Cooley and Lohnes model was intended to provide a theoretical framework for . ' 
evaluative inquiry. As such, it was oriented toward assessment of group rather than individual processes 
and outcomes. Their model retained the "opportunity" component of the Carroll model, but did not retain 
the emphasis on "time to learn." For example, "time spent" and "time needed" were not retained as 
intervening factors. ^ ^ 

Bloom's (1976) extension of Carroll's scheme has direct relevance to the AIS at Lowry. The variables 
in his paradigm consist of three major components: (1) cognitive entry behaviors, determining "the extent 
to which the student has already learned the basic prereqiiisites of the learning to be accomplished"; 
(2) affective entry characteristics,.determining "the extent to which the student is (or can be) motivated 
to engageTn the learning process*'; and (3) the quality of instruction, which indicates "the extent to which 
the instruction to be given is appropriate to the learner." All these factors interact on the task to he learned 
to detexmineHie nature of the learning outcomes: the level and type of achievement, the r^te of learning, 
and affective outcomes. * . . 

- ^ BloomTs primary thesis with regard to his model is that both student characteristics and quality of 
instruction can be modified to achieve a higher level of learning for individuals and groups. Quality of 
instruction can be evaluated by the qualities of cues, participation, and feedback in instruction. Bloom 
emphasizes the use of feedback and corrective procedures as on^ way of ensuring a high quality of instruc- 
tion. Furthermore, Bloom provides evidence that "gives support Jo.a strong inference that quality, of 
instruction has an effect on the learning processes of students as well as on their learning outcomes" 
(p.l35). . 

Specifically .with respect to learning rate. Bloom states that "when students are provided with the 
time and help they need to learn and when this produces positive entry characteristics (cognitive and 
affective), students not only become better able to learn, they also become able to learn with less and less 
time" (p, 191). Bloom cites results from a number of studies of mast'ery learning (Block, 1970; Arlin, 1973;. 
Anderson, 1973) as evidence for his claim. 

^fc notes that a student's learning characteristics can be altered positively or negatively at practically 
any point in a student's history, but that the potential for positive change is highest on the learning tasks 
that are early in a series. Thus, Bloom's model includes an interaction component between a student's 
learning characteristics and the quality of instruction, A higji quality of instruction, meaning the use of 
feedback and corrective procediire^^ early in a sequence of learning tasksT:an, according to Bloom, improve 
student e|ficiency in subsequent tas)cs and can reduce the variation in learning rates as well. ^ 

Past Evaluations of Mathematical Models of Student Progress • 

Of the three mathematical models presented earlier, neither Kcklin nor Chant and Luenberger 
present evaluations of their models with regard to descriptive or predictive adequacy. On the other hand, 
Suppes and his associates have conducted a number of studies of the trajectory model. (Suppes, Fletcher, 
and Zanotti, 1973, 1975, 1976; Larsen, Markosian, and Suppes, 1977; and Malone et al., 1977)* . 

In the three references by Suppes, Fletcher, and Zanotti and the reference by Larsen, Markosian, 
arid Suppes, the goodness of fit of the trajectory model is assessed using data collected from a variety of 
student populations: 297 deaf students on a CAI mathematics curriculum (Suppes et al„''1973; 1976); 
69 American Indian children attending a Bureau of Indian Affairs school and participating in a CAI 
mathematics program (Suppes et al., 1975); and 42 Stanford undergraduate students enrolled in CAI in : 
elemeritary logic (Larsen et al.VT977). ' ; * ; . - . 



AFHRL-TR-78.7 



AFHRL-TR-78-7 



On all three sets of data, the authors feel the model gives an adequate fit to the data. The most 
rtant parameter in fitting the data is the power factor k. Under the assumption that the power factor 
' k is constant across students, the goodness of fit is about the same^ for a rather broad range of values of k. . 
.TJie-fi^^ends tb .b^ substantially improved by allowing k to vary across students' . ^ ^ ' ^ \ 

In the first two studies cited, the distribution of k across students is flat o^ even U-shap^, showing 
a great deal of variation across students. In the last study, on the otherhand^the . distribution is concentrated 
in a short interval. . ' • .• \ :• : 

For k fixed, the correlation between the other two parameters, b and c, is very low. When all three 
parameters are estimated, however, high correlations among the three are found. This last result is not 
surprising since. changing the value of khas the effect of changing.the scale as well as the shape of the 
trajectory. , 

V Some results on using the '^trajectory modql forprediction. of pe 
. article by Larsen et al. (1977) and further results are con tallied in an. unpublished manuscript by Malone 
' et al. (1977). Larsen et al. (1977) examine two forms of the trajectory model to predict course completion 
time using performance times on initial lessons. One form requires'that all three parameters be estimated; 
the other form assximes the .power factor k to be given and fixed across students and only requires individual 
estimation of the remaining two parameters. For the initial third of the course that consisted of a total of 
30 lessons, the model assuminga fixed k provided substantially better course completion time predictions 
•than the model ttiat required k to be estimated for each individual student. For the remainder of the 
course' both models performed about as well and provided what the authors regarded as good predictions. 

In their unpublished manuscript, Malone et al. examine the. ability often alternative trajectory 
models to predict final grade placement, given time on a CAI drill and practice progriam in reading and 
mathematics. The ten mcw^els differ with respect to (1) assumptions regarding the pQwer factor, k; (2) 
assumptions regarding use of performance information consisting of grade placement at interim points in - 
the course; (3) assumptions regarding use of initial grade placement; and (4) assumptions reg^>ding how to 
estiniate the learning rate parameter, i). Data from approximately 3,000 students in third through six± 
grades in the Fort Worth Independent School District were used in the anialysis. The standard error of the 
difference between observed and predicted grade^lacement is used as the goodness of prediction measure. 
The authors find that the simplest two of the ten models^ gives the best predictions. These are models that 
assume that gain in grade placement is linear in, time (or the squa/e root of timie) on the system, rising from 
the last observed grade placement at a rate that is estimated from the population average. The author's 
explanation of the result that simpler models give better predictions i^ that the prediction was for a point 
outside the range of observations! 

• - . - ■ , ^ '* ■■ ' 

the more parameters that are available to fit the observations, the more sensitive the. . 

curve is to sniaU random fluctuations ill the data, and th^ it . 

can be wrong outside the range of the data. (Malone et al., 1977) 

*■ ■ ■ . • ■ ■ ■ - - ' ■ ■ '' ' ' * 

<■■' Other Studies of Student Progress ; • • 

Other studies of student progress have been based on Correlational aind regression analyses to identify 
factors explaining student rate of learning or to generate prediction eqimtions. A regression analysis may 
b.e considered to be a mathematical model in'the sense of hypothesizing a linear relationship between the 
rate of Jeaming or the course completion time and certain independeht variables. Such an approach 
corresponds to the milestone model specified in the prbposal and evaluated in the body of this report. 
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- A number of studiefs conducted by the learning Rejearch and Development Center (LRDC) at the 
University of Pittsburgh (Wang, 1968; Wang, 1970; W^iig and Undvall, 197 have examined the relation- 
ship between rate of learriing'anc^such variables as pupil.aptijude and achievement,. The studies analyzed* . . 
several sets of data collected on elementary school students who were participating in the Individually 
PrescriUJed Instruction Project (IPI) conducted by LRDC. Four alternative rates of progress were formulated 
The rates expressed progress in terms of point gain pn.tests, number of pages worked, and number of skills 
learned per unit of time. Independent variables included measures" of ajititude, academic achievement, and- 
prior classroom performance. ' * ' ■ " - 

Correlations between the rate measures and independent variables were generally very small, the / • 
largest conelations being in the range between .2 and ,4. The results of multiple regression analyses^and 
canonical correlation analyses indicated that there was some relationship betw^n rates and the independent 
variables. For example, Wang and lindvall got multiple correlation coefficients m the ranjge between ,34 
and .64. However, the regression coefficients were not consistent across data sets. Wang (1970) concluded 
from the inconsistency in the results that "rate of learning is specific tp a given task, and is not a general ^ - 
factor characterizing student perfoimance in an learning sit^ 

- Another study conducted at the LRDC and reported hy Meager and Kissel (1 969) confirms the . • 
importance of using variabfes related to the task in attempting to predict cornpletibn times* Data on ' 
between 63 and 69 elenientary-schbol students in eight different units of a mathematics program were used. 
Days needed to master the imit was the dependent variable an^ five independent variables were selected 
based on hypotheses concerning the process by which a tocher might develop a student's prescription* 
The'study found that between 52% and 71^ of the variance in coinpletion times on given learning units 
could be accounted for by the .five variables selected for-fetudy. A studeritVnnit pretest score ^d the 
number-ofesldlls he would have to master in a given uiiit were the .two best^^predictors of time^ ta master 
a given xmit. Age was a consistent, although less strong, predictor* IQ and number of units previously 
mastered were found to be relatively poorprfedictors. Thus, variables that were the most closely related 
to learning the required tasklwere the best predictors of completion times, ^ , . . 

A study of Wagner, Behringer, and PattieX1973) on individualized course completion time predictions 
was very similar to the present study. Th^ objective of the^study was to accurately predict each student's . 
course-completion date prior to graduation, for a U.S, Army Stock Control and Accounting Specialist 
cburse'Aat wais being converted to an individualized curriculum at the tinje of the study. 

Their literature review led to two conclusidns that they used in their approach to the problem: (1); 
measures of aptitude directly relevant to the course are better predictors of completion time than general . 
aptitude rneasures: and (2) the best prediction^equation of course completion time would be linear in the 
independent variables/ " ^ ' 

In the first phase of the sj;udy, availaWe predictor vaiiabl^^ 
Forces Qualification-Test (AFQT) and the Army Classification Battery (AC j3) of tests were correlated ' 
with time to completion and performance scores on sections of the conventional course that had been / . 
self-paced. With betweJ'eii 61 -and 77 students included in four separate analyses, the highest correlation 
with time to criterion was -,54 acWeved by the Arithmetic Reasoiiing test in ^ 

During the second and third phases of the study, a test battery was developed-that mesisured skills 
and kno\sdedge relevant to the specific course. Prediction equations were developed using a istepwise 
regression analysis with instructional time and total^course time as dependent variables. One of the ^ 
major findings was that by grouping students according to mode of instruction, either audiovisual (AV) 
or programmed instruction text (PI), a substantial improvement in prediction wa|;achieved. The multiple 
conelations corresponding to the final prediction equations using only baseline variables were .65 and .74 
for the AV and PI students, r^pectively, with 52 and 81. students, respectively. When.within-ccurse' • * 
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r . ■ \i - ' . . • 

performance times were included in the prediction, equations, the multiple regressions increased to about 
J&5 for both groups. The baseline variables that entered both sets of equations were the^score on the ' 
AFQT and scores on several course-specific tests. ' ^* . 

Review of Incentives and Inteivention Strategies ' ' ' v . * 

' ' • '■ ' • ' ' ■ - 

A significant aspect of Air^ Force technical training is the overall motivation on the part of trainees' 

*to pursue their^course^of study in an efficient and-dedicated.manner. Many trainees will be adequately 

* motivated by a desire to leamv the old maxim, *'leaming is its own reward," appears to have a rjcal-world 
basis. For these trainees, improving the quality of instructional material, instructors, and educational proce- . 

- dures is the key to minimizing their course completion times. For ^pinees without some minimum of 
motivation, the training course will take longer and be less effective, even if the instructional material and 
instructors are of high quality. Therefore, improving motivation is an important aspect of producing more 
efficient training. ^ ^ ^ ^ • . 

The literature on this general topic is very large, much too large to permit a comprehensive revie>y. 
Fortunately, however, the Air Force sponsored several recent research "projects fociised on the specific 
problems of motivation related to the Lowry AIS courses ^ritchard. Von Bergen, and DeLeo, 1974; 

• Klimoski, Raben, Haccoun, and Gilmqre, 1974; Raben, Wood,Klimoski and Hakel, 1974,Wood, Hakel, 
Del Gaizo, and Klimoski, 1975). These projects include comprehensive reviews of the ^terature, analysis 

V of attempts to use incentives to enhance motivation'of trainees, and'general conceptual analyses of 

- motivation in technical training and education. Since numerous studies have found that the best approach 
to predicting learning rates and to conducting aptitude-ireatment analysis is one tailored to the particular 
instructional setting (Cronbach and Snow, 1977; Wagner, Behringer, and Pattie, 1973;Packard, 1972; Wang, 
1968; Yeager and Kissel, 1969), it seems reasonable that prediction of responses to incentives needs to be * 
considered in a specific setting as well, and that the Air Force reports deserve special emphasis^ . ■ 

One of the Air Force reports is simply an annotated i)iIiogr,aphy of 234 references (Klimoski et al., 
1974). The companion report is an analytical review of the literature (Raben et al., 1974), the niajor cori- 
' elusion of which is that social reinforcement is related to a large number of "moderating** variables in 3ji • 
' extremely complicated manner.. This makes it impossible, at the present time, to predict the effects ofL- 
social reinforcement in a training situation. However, it is worthwhile to experiment with social reinforce- 
ment because of its relatively low cost and its effectiveness >yith at least some individual^. ^ . ^ 

- % One empirical study (Pritchar d et aL, 1 974) was an experiment using three different incentive motiva- 
tion systems: high-feasibility\(Le., cheap and easy tojrtiplement) incentives based on performance (magni- 

; ttide- of block scores); high-feasibility incentives based ^n effort (behavior in .the course), and high- and low- 
feasibility incentives based on effort. Pritchiard et al. made a fairly comprehensive review of the literature 

:and interviewed traineo^ instructional staff,^ and' administrators in two technical cburses at ChaniJte AEB 
to id^tify relevant incentives, implemented the three incentive systems, and analyzed the effects of 
incenuves on trainee performance. The niajor conclusion was that only the incentive system including' 
low-feasibility (i.e., expensive or difficult to arrange) incentives was cost-effective. The study suggests the^Jp 
foliWing^delines: ^ V ^ 

iX) Incentives must be fairly powerful: **Every attempt should be made to use incentives 
such as choice of assignment, promotion, and extra leave*' (p. 214). 

(2) Incentives are not cost-effective in courses where students are already performing 
near capacity. " ^ . 

(3) Self-paced courses are most appropriate for incentive techiiiques. 
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(4) Frequent reinforcement should be scheduled: opportunities to earn points should occur 
at least once a day. / , " . v . . 

(5) Both authority figures with vv^om th^ student comes in daily contact and his peers should 
provide positive social reiriforcemeht for high performance. 

. V Johnson, Salop, and Harding (1972) report two analogous studies of student responses to incentives ^ 
' a^d disincentives in an aviation mechanical fundamentals course utilizing CMI materials* They found an ^ ^ 

average*time saving of 11% over controls when a low-feasibility incentive --choice of Service Ratmg course 

after graduation — was used. 

Prediction equations developed from Navy Basic Test Battery scores'and times from previous classes 
were used to predict completion times for each of the experimental students- They were told that the order 
in which they could choose from among available Service Rating courses would depend on the ratio of 
their predicted completion times te their actual completion times. They were also^ven the disincentive of 
a Saturday morning study session if they lagged too far behind their predicted progress rate, but this was 

never applied because no student fell Very far behind his predicted rate. 

• * . ,• ■ . ~ . • 

Both the experinie^ital group and the control group were given the incentive of afternoons and 
.evenings off for whatevejr days remained between the day they completed the course and the day the 
course of ficially ended/Leaves away from the base were not granted. The time allotted to the course was : 
the same as that allotferd to the-same course taught by traditional methods; the aim of the incentive ^ ^ 
program was evidently to minimize thejinrhber of CMI students who took longer than the allotted course 
time, because no provision was made for assigning students to their initial duty station immediately if they 
finished the course early. , - . ' ^ . . 

The incentive. students (expeririientals) -did norc^fer si^ficantly from final 
tests when scores on the Arithmetic Reasoning Test (their best single predictor) were used as covariates to 
reduce within-group variability. They also did not differ markedly in attitude toward the course and toward 
the quality of their work. They did differ significantly in course completion time, avera^g 11% less time ^ 
than the copt^l group. Choice of (or at least having some controbover) one's training specialization and^. • 
location of duty assignment thu^seems to be a very effective incentive. • ^ * 

Johnson et al. do nptMiscuss'how well their predictive equations fit\actual student times in pa^ - * 
courses. They also do not discuss the circumstances and motivation of students taking the Navy Basic Test 
Battery. Since it. would be. advantageous.to a student to do^ poorly on the aptitude tests, .^ven the • 
incentive criterion used in. the course, thought shpuia' be given to motivating Svety student to do his best 
on the preassessment tests.as well as in the course. If this is riot posable, perhajte pifeinduction academic* 
performahce should be used to weight preassessment scores, or a progress criterion wei^ted to some > 

degree by all past students' rates should b^ used/ * : , ^ 

<p . ' - ■ ' " ■ . 

The last of the highly relevant Air Force reports (Wood et al^ i975) concerned the identificati6n of 
• incentives and analysis of them' in terms of feasibility, attractiveness, and other characteristics. A list of 
incentives was developed through interviews, lit er-ature review,'and group meetings: This lis^^ evaluated 
and refined through survey procedures with trainees and instructors. 

The more attractive incentives involved an effect directly on the trainee, for example, some choice 
in duty assignment or immediate promotion upon graduation. These tended to be quite costly or low in 
administrative feasibility, but their potential strength is supported by Katz's (1971) finding that many 
Navy recruits were highly motivated by upward social mobility when they signed up, and by Johnson, 
Salop, and Harding's finding that choice of specialization course and choice of initial duty assignment 
(even a choice between East Coast and West Coast) were the most valued of the available incentives. 
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' ' ' ^ ■ ' ■ 

Wood et al. found that Black trainees were more likely to prefer recognition-type incentives and 

White trainees were more likely to prefer control- and future^areer-ohented incentives. They conclude with 

a proposal to experiment with four incentive systems: v 

(1) Incentives administered by the instructor based on trainee pefl^^^ce. ' ^ - 

(2) Incentives administered by the instructor and the class as a^otip based on individual 
; • performance. . ' ^ ^ 

(3) Incentives administered by the instructor to^e individual based on the performance 
of the class as a group. ^ 

(4) Incentives administered by the instructor and the class-based on performance of the 
class. 

These four systems vary along the need-related or "dynamic" dimension as defined by Bond (1971), 
and do not provide for comparisons or study of interaction of task-related C*intrinsic") incentives and 
extema^motivatorsv(re wards). Thus this experiment does not permit evaluation of rewards that combine 
need-related incentives with external motivation, e.g., promotion in rank for honor students. Nor does it 
address Bond's evidence that effective external motivators reduce task-related, intrinsic satisfaction. Lx)wry 
course managers concerned with cost-effectiveness will want to weigh the greater power of various external 
. motivators against the lower cost of task-related an^ some need-related incentives. 

Among the many studies of traits that might respond to need-related incentives are those of 
Spieiberger et al. regarding anxiety in students. Spielberger, O'Neil; and Hansen (1972) found that students 
experiencing hi^ anxiety states tend to make more errors and perform less well on creative tasks thaii 
students who do not feel threatened. Spielberger . et al. cite studies which found that computer instruction 
lessens stress, on anxious students and enables them to perform better. However, an incentive system that 
included disincentives or competition for greatly desired rewards, particularly socially oriented rewards, ' 
might raise thdr anxiety levels to a point where their progress might be hampered. Even low-anxiety trait 
students, wMmay possibly perform better under slight pressure Spielberger et al., 1972), may do poorly 
if they are low in ability and "the task is taken seriously" (Cronbach and Snow, 1977, p. 398). 

Stress might be particularly likely if course materials are inadequate. According to Jamison, Suppes, 
and Wells (1974) a study made by Shrable and Sassenrath (1970) found "that an easy program with short 
steps is better suited to persons who are low on need for 'achievement and high on fear of failure or test 
anxiety, and that a difficult program ^feth long steps is preferable, for those with a higli need for achievement 
and low fear of failure/' Howevef, Tobias and Abramson (1971) failed to replicate this anxiety finding, : . 
Cronbach and Snow (1977) summarize a number of experimental studies on this topic by saying that it 
is not yet "under control**. All that can really be said about anxiety and need for achievementJsvAat they 
.may affect student progress differentially when a particular incentive system is introduced, and they*^ ^ 
should be monitored in case, the incentive .system would work better if tailored to this aspect of a trainee's 
personality.- ^ . . / . . 

Innovations To Stimulate Responsbility 

Another way to view the problem of motivation in a self-paced course is to consider how to stimulate 
self-responsibility for one's own leamingxn.a student who is not used'to having any latitude in his rate. 
Gordon (1970) suggests a gradual transfer from the teacher to the students, o£ responsibility for what, 
•how, and how fast students learn. ' * ' \ " ^ ' ; 
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Other techniques that may promote more responsibility in studients in a self-pacedlearning . : 

environment have been explored by several researchers. One of these is "micro teaching" or "peer tutoring" . 
in which a student who has mastered a skill assists a peer in learning it. Colton (1974) found that the peer 
tutor was able to do this at no cost to his own performance, although he'may not have included time as* a 
consideration. Sloan (1970) found that coUege students successfully counseled other college students on . 
academic, social, and personal adjustment problems. This could be extended to having a student counsel 
another one in techniques to* increase his rate of progress. In one of the Lowry incentive studies,: the 
chance to tutor a classmate was viewed as one of the more desirable high-feasibility incentives. Of course,* 
a strongincentive q^stem might work against the willingness of faster students to serve as tutors, unless it were 
structured to foster helpfulness. (The same reasoning would suggest that cheating might become a problem 
with a strong incentive system. )From a cost-effectiveness point of view, it would be necessary to test 
whether enough improvement in student times occurred 16 offset the cost of the program. Cohen and Fish- 
bein (1976) report success in traxjoing company commanders to have different behavior intentions using CAI, 
and this technique (a CAI guidance program) could help the student identify and alter his attitudes toward 
learning. Cogswell (1 966) counseled students regarding their performtoces via computer. A similar program 
could coach the ^dent in techniques for making rapid progress in the course. Such programs could conceiv- 
ably replace at least some of the counseling sessions specified in the current SPMS for students who fall behind. 

A type of intervention strategy that was not included in the Air Force studies is giving the student 
information on the average or top performance times of past students on a section of a course. This could 
provide a standard by which the trainee could gauge his own learning rate, and could be given to him either 
instead of, or along with, the prediction charts based on their presa^essment scores that are presently used. 
Colton (1974) gave students in one section of his self-paced college media course the average completion • 
times of past students on each unit, telling them to treat the information as "a possible guide to determine 
how efficiently they were usirr- rheir time" (p. 284), The experimental section averaged less time* than the 
control section on 18 of tlic 22 tasks; Uo /A^ver, they averaged more time than the pilot group whose 
i^rage times they hac been told. The format, aims, subject matter, and students in this class were different 
from those in the Lcwry courses, and it as n^ a tightly controlled experiment, so comparability is 
uncertain: In a stud^ of -^aval personnel learning :o use complex control systems, Myers (1969) found that 
giving students iiifcrmauon on post-traLiing performance times of course graduates definitely improved 
their speed. Again, drcumstances were differe^^t from the Lowry courses, and transferability is uncertain. 

Jeel(l967) reporting the use of a contract zpproaqh in an electricity-electronics course found that 
sorrie students. responded well to this treatment aad some did not, feeling that thejr needed more guidance 

: in acquiring fundament^sOjFace-to-face meetmgs setting up cpntractsx>n a smaller scale, e.g., overruling a 
lack of a prerequisite skiii or completing a bloc^of lessons by a certain date - might help son 
students to manage their time better It might possibly improve class times if contracts were i|sed for all 

^ students rather than pnly as a counrxling tool for those who fall behind. 

Other techniques that have been used. include team learning and varied presentation of m'kerial, 
where the student chooses how he wishes to be taught a given section of the course. 



^ Methods to Improve Course Material . ^T" : 

Finally, it ^ould be noted that good course material can stimulate an interest in learning and maxi- ^ 
mize progress, and improving instructional materials could .conceivably lead to significant time savings. 
This can be done rigorously, using, a computer model of the state of knowledge of the student at any point 
in.the- course, as is being attempted by Self (1974). It could also>be done (through experimentation) by 
defining'an aptitudertreatment decision network to evaluate student progress at many checkpoints and • 
provide the lemming conditions most suited to has current state of knowledge , ability, and personality 
(Schwen,!!l973X Parr (1973) describes three sophisticated CAI^progranJs that combme thorou^ analysis of 
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the subject*s structure with flexible presentation. These approaches are expensive and may not be achievable 
with some types.of course material. A more practical approach to improving the course might be to 'elicit 
detailed feedback "from students who encounter difficulties in the course and from people who work, closely 
with students. Also, records of student times on small segments of the course could be analyzed to pinpoint 
bottlenecks. Perhaps counseling sessions could include discussions of what changes in the course might help 
student progress. This emphasis in counseling sessions might improve the counselors* perceptions of their • ^ 
' role and make them more likely to follow procedures. Also, typical gaps in the entering students* back-] 
ground could be identified - perhaps with ".mini-lesson" tests such as those implemented by Wagner et al. 
(1973) ~ and suitable precourse remediation provided; experimentation would deterrhine whether such 
remediation improved course completion times. Anderson (1976) found in a study of 90 eighth-grade stu- 
dents that "A group of students enter a particular learning sequence with unequal amounts of relevant 
prior learning*'' but '*by complementing inequality in learner characteristics with inequality in instructional 
time and help in the early units, we can approach student equality in later units, not only in the. achievement 
level attained, but also in. the amount of on-task time needed to attain the criterion level*' (p. 233). Struc- 
tured observation of student behavior in the classroom has been used by Spielberger et al. (1972) to identify 
portions of a course that produce anxiety in students, and by Yeager and lindvall (1968) to evaluate instrud- 
tional innovations. Finally^ course material can be examined to see whether it follows sound educational 
principles such as eliciting active behavior from the student (Suppes, 1964), providing quick feedback . 
remediation with "enough information for the student to diagnose his owii shortcomings^.' (Rosenbaum 
1969, p. 3), and avoiding ambiguity, excessive repetition, and requiring unrelated skills. . . 

Conclusions . • . " ' ^ - > 

^' : /Based on the review of the literature on mathmatical models of student progress and on motivation 
and intervention strategies, the following conclusions were made: " .].-.> 

• There are v^ few global^athematical rnodels of student progress. 

•> The mathematical models t&t have teen found to be most successful at prediction have been' 
, ^ those with relatively simpre structure.- . ' - ^ * 

/ • The accuracy of prediction of student progress ificreases as a function ofthe relevance of the , 
' T . . predictor variables to the learning task: ; / . 

' • Powerful exte^ai incentives improve sptudent performance; intrinsic (task-related) or social . 

incentiveshavelessef^trbut are generally easier to implement becaus% of lower cost^ ' 

• ' • .^Spnie powerful externai incentives may displace and others augment the effect of 
. intrinsic or social incentive^ when both are present. A 

• Response to incentives varro$-by the demographic characteristics of the students, such 
as' age arid socioeconomic status, as well as by aptitude and prior knowledge. It may also 

vary depending on measurable personality traits. ,v • 

' • Aside from incentives; rate of learning can be increased by improving instructional material 

and the quality of instruction, and by fostering self-responsibiKty through innovative approaches. 
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> Appendix B ^ 

SUMMARY STATISTICS 

For purposes of reference, the summary statistics for the data included in the analysis are shown in 
Table B-l and B-2. Table B-1 shows^the means and"standard deviations for the baseline variables cumulative 
. block elapsed times, and selected cumulative lesson elapsed times' Table B-2 gives the correlations Miong 
the variables included in the analyses. 



/ 



ERIC ' 



58 



AFHRL-TR-78-7 



AFHRI.TR-78-7 



. Table B-1 

MEANS AND STANDARD DEVIATIONS FOR VARIABLES 
INCLUDED IN THE EVALUATION 
(n = 368)« 



Variable Typcr 


^ Variable Title . 


Mean 


S.D. 




Sex ^ 


(79% male) 






Hicyhe^t^^chnol Year Comoleted 


12.4 


.9 




Aop ?»t r^niir^ Pntrv 


20.9 


2.6 


<• 




133 


5.1 




Readinff Vocab Scientific Scale 


8.0 


2.9 




Reariinc'Vocah Total Scale 


213 


7.4 




Pre-Course State Curiosity 


64.1 


8.8 




Pre-Course State Anxiety . 


37.4. ^ 


8.9 




Trait Curiosity ^ i ' ^ 


25.5 


5.8 




Trait Anxiety ^" 


• 36.4 - 


. . 8.8 




Internal -External Scale 


; 14.6 


' 43 : 




Text Anxiety 


30.0 


7.4 . 




Prefer, for Audio Mode 


' 10.7 


2.6 




Prefer, for Visual Mode 


6.8 


/ 1.8 




Prefer, for Printed Mode . . - 


- 4.9 


1.4 


* * 


Experience with Self-Pacing ^ 


5.5 


" 1.4 




Experience with Convent. Instru. 


' 6.8 


1.5 




- IM/MF'Readine Subscale 1 


- -43 ' 


1.6 

\ 2.1 




IM/MF Reading Subscale 2 


6J 




IM/MF Reading Total : 


103 


'3.2 




IM/MF Logical Reasoning Scale 


. 203 


7.1 




Concealed Figures Scale 


5.S 


2.7 




- Memory for Numbers Backward Scale . 


18.6*' 


• 2.8 


- s ■ ■ .-. " 


Memorv for "NumT^ers. Total Scale 


' 353 


5.0 


Cumulative Actual 


CABELT 1 ■ 


1219. 


439 


'Block Elapsed 


. CABELT 2 


2915 . 


875 


'Times** . - 


GABELT3 


' 5210 


1449 • 




CABELT4 


6601 . . 


1726 




CABELT 5 


82iB 


2038 r 




CABELT 6 


9405 - 


> 2193 


Cumulative Lesson 


CET9(Brock 1) 


1033 . 


336 


Elapsed Times at 


CET19 (Block 2) 


2378 


■ 677 /. 


the End of Each 


CET31 (BIock3) 


/ 4025. 


1073 


Block ' 


CET40 (Block 4) 


' 5063 


1311 




CET52 (Block 5) 


6250 


1504 ] 




CET61 (Block 6) ^ 


7186 


1640 -> 



Students in course vetsion 1 with all blocl^lapsed times reliable i^d a course entry date in 

of absence times. - I- * 



**Simi of block elapsed times minus.th 



-57-59 
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Table B-2 . 

CORRELATIONS BETWEEN BASELINE VARIABLES AND 
CUMULATIVE TIME VARIABLES AND AMONG 
CUMULATIVE TIME VARL\BLES 
(n = 368) 

' ■ , . « 

Variable 



Label 


CAfeELTl 


CABELT2 


CABELT3 


CABELT4 


CABELT5 


CABELT6 






.Z / 


.Z / 


97 


9^ 


9^ 
*Zj 


• 111 I tlAlVV 


.Ul 




•Ul 


nn 

.UU 


•UU 


.Ul 






.Uo 


.uo 


nc 

.UD 


n9 


no 

.U3 






— / 


— / 


97 


97 


9< 

—.ZD 






— 


9A 


9< 


97 


9< 


KV Ui^ ILr 


— .z^ 


— .ZO 


90 


9Q 


9Q 


97 


■ CTPT fD 

olLUK 


c\n 
Xjj 


.UD 


.Uz 


no 
.U3 


n 1 
.01 


no 
.U2 




.Uo 


.0^ ^ 


,11 


1 n 
.10 


in ' 
.10 


^ in 
.10 


iKLUK 




.Uo 


.U7 


.07 


n^ 
.06 


n^ 
.Oo 


IKAINA 


.03 


.UU 


no 
,02 


.03 


.04 


.04 


lESCL 


-.04 


•".13 


-.17 


-.17 


-.18 


-.17 ^ 


TSTANX 


.13 


.17 


,17 


.18 


.19 


.18 


PREFA 


n? 


.00 


.01 


.01 


.01 


.00 


PREFV ^ 


,01 


.03 


A A 

.04 


.03 


.03 


.03 


PREFP 


. -.04 


.05 


.06 


.06 


Sio 


.06 


EXPSP 


-.08 


.10 


* .11 


.12 


.11 


,12 


EXPCI 


-.18 


-.25 


-,27 


-.28 


-.29 




READS 1 




-JO 


-JO 


-31 


-32 


-32 


READS2 


-.28 


.-J3 


-33 


-32 


-3l 


*5n 

-30 


READST 




—,37 


-.37 


-36 


—,37 


-35 


. LOGREA 


-.23 


-.34 


-31 


O 1 

-.31 


on 

-30 


on. 

-30 


CONrlu 


-.05 


-.10 *■ 


-.10 


-".11 


' t n 
-.10 




MJbMNB 


.05 


-.09 


-.16 . 


-.15 - 


-.14 


- J3 


McMNT ^ 


.02 


1 A 

—.14 


'^n 
-.20 




—.17 


—.15 


CABELTl 




.87 ^ 


.76 


.75 - 


.72 


.71 


CABELT2 


■ .8?" * 


■ - j 


.93-. 


. ,91 


.88 - 


.86 


CABELT3 


.76 


■ .93. , 






.96 


.94 . 


CABELT4 


' .75 


.91 


.99 


•98 


.97 


CABELT5 


:72: ■ 


-.88 


:96^ • 


.98 




.99 


CABELT6 


.71 


.86^ 


.94 


.97 


. .99 




CET9 


,80 


V .80 


.73 


,72 


.70 


.69 • 


^CETJ9 


.73. 


\.84 . 


" ;8o 


.79 


/ . ^78 . 


.76 


^CET31 


.69 


, .80 


.82 


.83 


■ ,80 


.79 


CET40 


'\68 


.78 : ■ 


; ■ -SI 


; .83 - 


.81 


" ..80 


CET52 


' -68 


.- ' .77 " 


.80 


. ,83 


.83 


.83 


CET61 


. .66 . 


-.76 ^ 


.79 


.82 


■ .83 


. -.84 
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Table 



Variable' 



Label 


- CET9 


CET19 


CUV 


• 11 




UTVI? A O 

nlitAK 


-LHJ 






.Ud 






on 


OO 




-•25 


o c 


KVUCi L 




— .JU 


STCUR 


•Uo 




STANX' 


•11 


1 o 
•J 2 


TRLUK 


.10 


no 
T .Oo 


TRANX 


.01 


.01 


lESCL 


-.09 


—.13 


TSTANX 


.19 ^ 




PREFA 


.04 


.02 


PREFV 


.02 


-.05 


Tin 1 'Ik 

PRErP 


Art 

-.09 


— .oy 


EXPSP 


-.15 


-.15 


Expcr 


-.28 


-.28 


READSI 


, -.30 


-.31 


READS2 


-.30 


-.29 


READST 


-35 


-.35- 


LOGREA 


-.28 


-.32 


CONFIG 


. -.10 


-.09 


MEMNB 


-.09 


-.09 


MEMNT 


-.09 


-.10 


(For GABELTs (see previous page) 


CET9 




.90 


CET19 


.90 




CET31 


-82 


.93 


CET40 


.78 


' / ■ '.89' . 


CET52 


.73 


■ • " .84 


CEt6I 


. .71 , 


.81 
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B-2 (Continued) 



CET31 


CET40 


CET52 


CET61 


23 ' 


.23 




.21 


-02 


-.03 


.00 


.01 


05 


05 


.05 


.05 


-2 7 


- 25 


-23 


-.21 


- 22 


- 20 


-.19 


-.17 


— .zo 


— .^v 


- 23" 


-.22 ' 




05 


04 

.V/— 


05 


in 


no * 


07 


06 

•V/VJ 


no 


no 


in 

.XV/ 


no 


nn ■ 




— n9 


' — n9 ' 


1 1 i 




— nR 


- n6 




•1 / 


1 < 

.1 .J 


1 9 r 


nn 


ni 




ni 


nA 


HA 
— .U*+ 


« n^ 


n^ 


— no 


— ,uo 


- n7" 


- 06 


- -.14 - 


-.14 


-.13 


■ -.13 


-.29 


-.28 


-.26 


- -.26 ■ 


• -.28 


-.28 . 


-.28 


-.28 


-.28 _ 


-.28 


-.26 


, —.26 


-33 • 


-33 


-32 


-31 


-.30 


. ^-29 


-;27 


-:26 


-.06 


-.06 


■ -.03 


-.04 


-.id" 


-.10 


-.08 . 


-.07 


-.12 ■ ■ '• 


•-.12 . ' 


-.10 . 


-.09 ' 








^ ■ 


-82 


.78 . 


.73 ., 


.71 


.93 


.89 ■ ^ . 


..84' 


.81 




.98 


^94 


.91 


.98 . 




.98 


.95 


■ -94 


■ .98 ■ 




: ,:99 


.91 


.95 . • . 


.99; " 
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